Jul 15 10:43:26.542947 ip-10-0-133-26 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 15 10:43:26.542957 ip-10-0-133-26 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 15 10:43:26.542964 ip-10-0-133-26 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 15 10:43:26.543187 ip-10-0-133-26 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 15 10:43:36.543150 ip-10-0-133-26 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 15 10:43:36.543167 ip-10-0-133-26 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e3392e062bdc40c9874931fb998532df --
Jul 15 10:45:29.429890 ip-10-0-133-26 systemd[1]: Starting Kubernetes Kubelet...
Jul 15 10:45:29.806146 ip-10-0-133-26 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 15 10:45:29.806146 ip-10-0-133-26 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 15 10:45:29.806146 ip-10-0-133-26 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 15 10:45:29.806146 ip-10-0-133-26 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 15 10:45:29.806146 ip-10-0-133-26 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 15 10:45:29.807414 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.807324    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 15 10:45:29.814526 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814505    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:45:29.814526 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814523    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:45:29.814526 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814528    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:45:29.814526 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814532    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814536    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814540    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814544    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814548    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814551    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814555    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814558    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814562    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814565    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814569    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814573    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814576    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814597    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814601    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814604    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814609    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814621    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814625    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814629    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:45:29.814851 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814633    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814636    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814640    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814644    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814648    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814653    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814656    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814660    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814664    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814668    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814671    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814676    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814683    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814690    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814695    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814699    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814703    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814708    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814712    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:45:29.815675 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814717    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814721    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814725    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814730    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814736    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814740    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814744    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814748    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814752    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814757    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814762    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814766    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814770    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814774    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814778    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814782    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814786    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814790    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814795    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814799    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:45:29.816463 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814803    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814807    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814812    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814818    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814822    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814828    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814833    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814838    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814842    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814848    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814853    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814856    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814863    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814873    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814878    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814883    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814888    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814892    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814897    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:45:29.817279 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814901    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814905    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814910    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814914    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814918    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814922    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814926    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814930    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814934    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814939    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814943    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814947    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814951    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814955    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814960    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814964    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814968    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.814972    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816188    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816198    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:45:29.818089 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816203    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816208    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816213    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816218    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816222    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816227    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816231    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816235    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816239    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816243    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816248    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816252    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816256    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816261    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816265    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816269    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816273    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816277    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816281    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816285    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:45:29.818951 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816289    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816294    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816298    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816302    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816307    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816310    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816315    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816319    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816323    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816327    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816331    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816335    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816340    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816344    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816349    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816353    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816358    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816362    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816366    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816371    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:45:29.819699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816374    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816378    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816382    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816386    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816390    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816398    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816404    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816409    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816414    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816419    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816423    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816428    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816433    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816437    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816441    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816447    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816452    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816457    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816461    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:45:29.820208 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816465    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816468    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816472    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816477    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816481    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816489    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816494    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816499    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816504    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816508    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816514    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816518    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816524    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816528    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816532    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816536    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816540    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816544    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816548    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:45:29.820784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816553    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816557    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816562    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816566    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816570    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816574    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816601    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816606    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816610    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816614    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816619    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816623    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816627    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816631    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816634    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816638    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816642    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816647    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.816652    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817288    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:45:29.821346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817296    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817301    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817306    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817311    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817316    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817320    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817325    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817329    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817333    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817337    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817341    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817345    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817350    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817354    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817358    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817362    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817368    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817374    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817379    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817383    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:45:29.821989 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817387    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817390    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817395    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817399    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817403    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817407    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817411    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817416    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817420    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817425    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817429    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817433    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817437    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817442    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817446    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817450    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817464    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817468    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817473    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:45:29.822468 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817477    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817481    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817485    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817491    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817495    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817499    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817503    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817507    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817511    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817515    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817519    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817523    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817528    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817533    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817537    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817541    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817545    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817549    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817553    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817557    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:45:29.823040 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817561    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817565    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817569    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817574    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817597    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817602    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817606    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817612    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817618    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817622    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817628    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817633    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817637    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817641    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817648    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817652    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817656    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817660    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817664    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817668    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:45:29.823524 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817677    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817681    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817685    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817689    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817693    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817697    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817701    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817705    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817709    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817713    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817717    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817721    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817725    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817729    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817733    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817738    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817742    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817748    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.817754    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817857    2566 flags.go:64] FLAG: --address="0.0.0.0"
Jul 15 10:45:29.824130 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817868    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817880    2566 flags.go:64] FLAG: --anonymous-auth="true"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817887    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817894    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817899    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817907    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817914    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817920    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817925    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817930    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817935    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817940    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817948    2566 flags.go:64] FLAG: --cgroup-root=""
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817953    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817957    2566 flags.go:64] FLAG: --client-ca-file=""
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817962    2566 flags.go:64] FLAG: --cloud-provider="external"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817967    2566 flags.go:64] FLAG: --cluster-dns="[]"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817973    2566 flags.go:64] FLAG: --cluster-domain=""
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817978    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817983    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817987    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817993    2566 flags.go:64] FLAG: --container-log-max-files="5"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.817999    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 15 10:45:29.824643 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818003    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818008    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818014    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818018    2566 flags.go:64] FLAG: --contention-profiling="false"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818024    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818028    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818033    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818038    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818044    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818048    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818053    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818058    2566 flags.go:64] FLAG: --enable-load-reader="false"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818062    2566 flags.go:64] FLAG: --enable-server="true"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818068    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818081    2566 flags.go:64] FLAG: --event-burst="100"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818086    2566 flags.go:64] FLAG: --event-qps="50"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818090    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818095    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818099    2566 flags.go:64] FLAG: --eviction-hard=""
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818113    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818118    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818124    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818129    2566 flags.go:64] FLAG: --eviction-soft=""
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818133    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818138    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 15 10:45:29.825227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818143    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818148    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818153    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818157    2566 flags.go:64] FLAG: --fail-swap-on="true"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818162    2566 flags.go:64] FLAG: --feature-gates=""
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818168    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818173    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818178    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818183    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818189    2566 flags.go:64] FLAG: --healthz-port="10248"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818193    2566 flags.go:64] FLAG: --help="false"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818198    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-133-26.ec2.internal"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818203    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818207    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818212    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818217    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818223    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818227    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818232    2566 flags.go:64] FLAG: --image-service-endpoint=""
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818236    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818241    2566 flags.go:64] FLAG: --kube-api-burst="100"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818249    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818254    2566 flags.go:64] FLAG: --kube-api-qps="50"
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818259    2566 flags.go:64] FLAG: --kube-reserved=""
Jul 15 10:45:29.825924 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818263    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818268    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818273    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818277    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818283    2566 flags.go:64] FLAG: --lock-file=""
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818288    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818292    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818297    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818306    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818311    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818315    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818320    2566 flags.go:64] FLAG: --logging-format="text"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818324    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818330    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818334    2566 flags.go:64] FLAG: --manifest-url=""
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818339    2566 flags.go:64] FLAG: --manifest-url-header=""
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818345    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818350    2566 flags.go:64] FLAG: --max-open-files="1000000"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818357    2566 flags.go:64] FLAG: --max-pods="110"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818361    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818367    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818371    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818376    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818381    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818385    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 15 10:45:29.826522 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818390    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818401    2566 flags.go:64] FLAG: --node-status-max-images="50"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818407    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818411    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818418    2566 flags.go:64] FLAG: --pod-cidr=""
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818423    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818432    2566 flags.go:64] FLAG: --pod-manifest-path=""
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818437    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818442    2566 flags.go:64] FLAG: --pods-per-core="0"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818446    2566 flags.go:64] FLAG: --port="10250"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818451    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818455    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09bbaeba8e855b15a"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818462    2566 flags.go:64] FLAG: --qos-reserved=""
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818467    2566 flags.go:64] FLAG: --read-only-port="10255"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818472    2566 flags.go:64] FLAG: --register-node="true"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818477    2566 flags.go:64] FLAG: --register-with-taints=""
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818483    2566 flags.go:64] FLAG: --registry-burst="10"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818487    2566 flags.go:64] FLAG: --registry-qps="5"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818492    2566 flags.go:64] FLAG: --reserved-cpus=""
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818498    2566 flags.go:64] FLAG: --reserved-memory=""
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818503    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818508    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818513    2566 flags.go:64] FLAG: --rotate-certificates="false"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818518    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818522    2566 flags.go:64] FLAG: --runonce="false"
Jul 15 10:45:29.827165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818527    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818531    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818536    2566 flags.go:64] FLAG: --seccomp-default="false"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818541    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818545    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818550    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818555    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818559    2566 flags.go:64] FLAG: --storage-driver-password="root"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818564    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818569    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818573    2566 flags.go:64] FLAG: --storage-driver-user="root"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818597    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818602    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818613    2566 flags.go:64] FLAG: --system-cgroups=""
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818618    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818627    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818631    2566 flags.go:64] FLAG: --tls-cert-file=""
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818635    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818641    2566 flags.go:64] FLAG: --tls-min-version=""
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818648    2566 flags.go:64] FLAG: --tls-private-key-file=""
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818652    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818657    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818661    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818666    2566 flags.go:64] FLAG: --v="2"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818672    2566 flags.go:64] FLAG: --version="false"
Jul 15 10:45:29.829013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818678    2566 flags.go:64] FLAG: --vmodule=""
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818684    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.818690    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818908    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818915    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818920    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818925    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818929    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818934    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818938    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818942    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818946    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818950    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818954    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818958    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818963    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818967    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818971    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818980    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:45:29.829702 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818986    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818991    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.818996    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819001    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819005    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819009    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819013    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819020    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819024    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819028    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819032    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819036    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819042    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819048    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819052    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819056    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819061    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819065    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819069    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:45:29.830221 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819073    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819078    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819082    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819086    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819090    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819094    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819098    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819102    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819106    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819111    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819115    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819119    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819125    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819129    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819133    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819137    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819149    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819154    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819158    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819163    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:45:29.830725 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819168    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819173    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819177    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819181    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819185    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819189    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819193    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819197    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819202    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819205    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819210    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819214    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819218    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819222    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819225    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819229    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819233    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819237    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819241    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819246    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:45:29.831207 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819250    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819257    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819262    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819266    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819274    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819278    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819282    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819287    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819291    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819303    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819308    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819312    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819318    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819322    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819326    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819330    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819335    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819338    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819342    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:45:29.831717 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819346    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:45:29.832195 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819351    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:45:29.832195 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819355    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:45:29.832195 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819359    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:45:29.832195 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.819363    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:45:29.832195 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.820518    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 15 10:45:29.832195 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.827295    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 15 10:45:29.832195 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.827309    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 15 10:45:29.832195 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.827325    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 15 10:45:29.832195 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.827331    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 15 10:45:29.832195 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827393    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:45:29.832195 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827399    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:45:29.832195 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827402    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:45:29.832195 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827405    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:45:29.832195 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827408    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:45:29.832195 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827410    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827413    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827415    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827418    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827421    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827423    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827426    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827429    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827432    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827434    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827437    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827441    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827444    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827448    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827450    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827453    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827455    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827458    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827461    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:45:29.832560 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827463    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827465    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827469    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827472    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827474    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827477    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827480    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827483    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827485    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827488    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827490    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827493    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827495    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827498    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827500    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827502    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827505    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827507    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827510    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827512    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:45:29.833042 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827515    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827519    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827522    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827524    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827527    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827530    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827532    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827535    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827538    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827540    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827543    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827545    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827547    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827550    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827553    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827555    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827557    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827560    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827563    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827566    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:45:29.833531 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827568    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827571    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827573    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827576    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827593    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827596    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827599    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827601    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827604    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827606    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827609    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827611    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827613    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827616    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827619    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827622    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827624    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827627    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827630    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:45:29.834166 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827632    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:45:29.834671 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827635    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:45:29.834671 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827637    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:45:29.834671 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827639    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:45:29.834671 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827642    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:45:29.834671 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827644    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:45:29.834671 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827647    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:45:29.834671 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827649    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:45:29.834671 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827653    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:45:29.834671 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827656    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:45:29.834671 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827659    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:45:29.834671 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827662    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:45:29.834671 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827665    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:45:29.834671 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827667    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:45:29.834671 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827670    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:45:29.834671 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827672    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:45:29.835053 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.827677    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 15 10:45:29.835053 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827788    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 15 10:45:29.835053 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827793    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 15 10:45:29.835053 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827796    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 15 10:45:29.835053 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827798    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 15 10:45:29.835053 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827801    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 15 10:45:29.835053 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827804    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 15 10:45:29.835053 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827807    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 15 10:45:29.835053 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827810    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 15 10:45:29.835053 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827813    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 15 10:45:29.835053 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827816    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 15 10:45:29.835053 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827819    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 15 10:45:29.835053 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827822    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 15 10:45:29.835053 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827824    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 15 10:45:29.835053 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827828    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827831    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827834    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827836    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827838    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827841    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827844    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827846    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827849    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827852    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827854    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827856    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827860    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827863    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827865    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827867    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827870    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827872    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827875    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 15 10:45:29.835457 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827877    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827879    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827882    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827884    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827887    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827889    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827892    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827894    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827897    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827899    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827902    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827904    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827907    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827909    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827912    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827914    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827916    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827918    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827921    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 15 10:45:29.836203 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827924    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827926    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827928    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827930    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827933    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827936    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827938    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827941    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827943    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827946    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827948    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827950    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827953    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827955    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827957    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827961    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827964    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827967    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827970    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827972    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 15 10:45:29.836917 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827975    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827978    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827980    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827983    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827986    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827988    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827992    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827994    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827996    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.827999    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828001    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828004    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828006    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828008    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828011    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828013    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828015    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828018    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828020    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828024    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 15 10:45:29.837555 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828027    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 15 10:45:29.838063 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828030    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 15 10:45:29.838063 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828033    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 15 10:45:29.838063 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828036    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 15 10:45:29.838063 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828038    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 15 10:45:29.838063 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828041    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 15 10:45:29.838063 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828044    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 15 10:45:29.838063 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:29.828047    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 15 10:45:29.838063 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.828051    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 15 10:45:29.838063 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.828158    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 15 10:45:29.838063 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.830033    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 15 10:45:29.838063 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.831353    2566 server.go:1019] "Starting client certificate rotation"
Jul 15 10:45:29.838063 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.831469    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 15 10:45:29.838063 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.831503    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 15 10:45:29.838392 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.834009    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 15 10:45:29.841809 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.841788    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 15 10:45:29.852328 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.852305    2566 log.go:25] "Validated CRI v1 runtime API"
Jul 15 10:45:29.854225 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.854205    2566 log.go:25] "Validated CRI v1 image API"
Jul 15 10:45:29.855022 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.855008    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 15 10:45:29.857530 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.857511    2566 fs.go:135] Filesystem UUIDs: map[12b98abf-31cc-41fc-bcf6-4d218e00f4c0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fc129fb7-767c-49c3-b8b4-26aa4ce5fa2f:/dev/nvme0n1p3]
Jul 15 10:45:29.857573 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.857534    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 15 10:45:29.862799 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.862686    2566 manager.go:216] Machine: {Timestamp:2026-07-15 10:45:29.861346865 +0000 UTC m=+0.318227537 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091750 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2acb97bb8c42e806150ea30cdc5cbf SystemUUID:ec2acb97-bb8c-42e8-0615-0ea30cdc5cbf BootID:e3392e06-2bdc-40c9-8749-31fb998532df Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9a:e2:78:19:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9a:e2:78:19:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:d7:c8:45:ae:a5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 15 10:45:29.862799 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.862796    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 15 10:45:29.862916 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.862872    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 15 10:45:29.865885 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.865857    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 15 10:45:29.866038 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.865890    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-26.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 15 10:45:29.866150 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.866133    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 15 10:45:29.866688 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.866676    2566 topology_manager.go:138] "Creating topology manager with none policy"
Jul 15 10:45:29.866728 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.866692    2566 container_manager_linux.go:309] "Creating device plugin manager"
Jul 15 10:45:29.866728 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.866700    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 15 10:45:29.867330 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.867320    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 15 10:45:29.868608 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.868597    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 15 10:45:29.871288 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.871277    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 15 10:45:29.871420 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.871409    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 15 10:45:29.871472 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.871440    2566 kubelet.go:486] "Attempting to sync node with API server"
Jul 15 10:45:29.871472 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.871452    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 15 10:45:29.871472 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.871472    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 15 10:45:29.871618 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.871481    2566 kubelet.go:398] "Adding apiserver pod source"
Jul 15 10:45:29.871618 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.871489    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 15 10:45:29.871860 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.871849    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 15 10:45:29.871927 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.871867    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 15 10:45:29.873948 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.873933    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 15 10:45:29.875472 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.875458    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 15 10:45:29.875541 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.875489    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 15 10:45:29.875678 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.875665    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 15 10:45:29.875718 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.875687    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 15 10:45:29.875718 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.875696    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 15 10:45:29.875718 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.875704    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 15 10:45:29.875718 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.875716    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 15 10:45:29.875821 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.875725    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 15 10:45:29.876207 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.876197    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 15 10:45:29.876240 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.876209    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 15 10:45:29.876240 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.876217    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 15 10:45:29.876240 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.876223    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 15 10:45:29.876240 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.876231    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 15 10:45:29.876240 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.876238    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 15 10:45:29.877509 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.877498    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 15 10:45:29.877544 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.877510    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 15 10:45:29.883209 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.883017    2566 server.go:1268] "Started kubelet"
Jul 15 10:45:29.883960 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.883875    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 15 10:45:29.884044 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.883989    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 15 10:45:29.884224 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.884184    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 15 10:45:29.884677 ip-10-0-133-26 systemd[1]: Started Kubernetes Kubelet.
Jul 15 10:45:29.885291 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:29.885111    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 15 10:45:29.885291 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.885141    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-26.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 15 10:45:29.885525 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:29.885318    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-26.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 15 10:45:29.885525 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.885365    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 15 10:45:29.887029 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.887012    2566 server.go:310] "Adding debug handlers to kubelet server"
Jul 15 10:45:29.890058 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.890038    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 15 10:45:29.890058 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.890048    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 15 10:45:29.890208 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:29.890136    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Jul 15 10:45:29.890611 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.890556    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 15 10:45:29.891191 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.890877    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 15 10:45:29.891191 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.890894    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 15 10:45:29.891191 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.891080    2566 reconstruct.go:97] "Volume reconstruction finished"
Jul 15 10:45:29.891191 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.891090    2566 reconciler.go:29] "Reconciler: start to sync state"
Jul 15 10:45:29.892510 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:29.890646    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-26.ec2.internal.18c270b61e0edb2d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-26.ec2.internal,UID:ip-10-0-133-26.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-26.ec2.internal,},FirstTimestamp:2026-07-15 10:45:29.882786605 +0000 UTC m=+0.339667278,LastTimestamp:2026-07-15 10:45:29.882786605 +0000 UTC m=+0.339667278,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-26.ec2.internal,}"
Jul 15 10:45:29.892907 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.892879    2566 factory.go:153] Registering CRI-O factory
Jul 15 10:45:29.892907 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.892899    2566 factory.go:223] Registration of the crio container factory successfully
Jul 15 10:45:29.893019 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.892948    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 15 10:45:29.893019 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.892958    2566 factory.go:55] Registering systemd factory
Jul 15 10:45:29.893019 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.892967    2566 factory.go:223] Registration of the systemd container factory successfully
Jul 15 10:45:29.893019 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.892988    2566 factory.go:103] Registering Raw factory
Jul 15 10:45:29.893019 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.893001    2566 manager.go:1196] Started watching for new ooms in manager
Jul 15 10:45:29.893440 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.893410    2566 manager.go:349] Starting recovery of all containers
Jul 15 10:45:29.893570 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:29.893539    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 15 10:45:29.903042 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.903006    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 15 10:45:29.903860 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.903836    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4427t"
Jul 15 10:45:29.905054 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.905037    2566 manager.go:354] Recovery completed
Jul 15 10:45:29.905548 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:29.905519    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 15 10:45:29.905697 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:29.905671    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-26.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 15 10:45:29.909908 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.909895    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:45:29.911809 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.911788    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:45:29.911863 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.911818    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:45:29.911863 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.911828    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:45:29.912328 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.912312    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 15 10:45:29.912328 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.912327    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 15 10:45:29.912427 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.912342    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 15 10:45:29.912427 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.912341    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:45:29.913042 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.913027    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:45:29.913108 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.913058    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:45:29.913108 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.913073    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:45:29.913224 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.913208    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4427t"
Jul 15 10:45:29.913838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.913826    2566 policy_none.go:49] "None policy: Start"
Jul 15 10:45:29.913886 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.913842    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 15 10:45:29.913886 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.913852    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 15 10:45:29.914918 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.914908    2566 policy_none.go:47] "Start"
Jul 15 10:45:29.949163 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.949142    2566 manager.go:337] "Starting Device Plugin manager"
Jul 15 10:45:29.949234 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:29.949210    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 15 10:45:29.949234 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.949223    2566 server.go:87] "Starting device plugin registration server"
Jul 15 10:45:29.949461 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.949449    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 15 10:45:29.949500 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.949463    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 15 10:45:29.949604 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.949573    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 15 10:45:29.949761 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.949745    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 15 10:45:29.949761 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:29.949760    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 15 10:45:29.950149 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:29.950131    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 15 10:45:29.950206 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:29.950184    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-26.ec2.internal\" not found"
Jul 15 10:45:30.034423 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.034391    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 15 10:45:30.035513 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.034434    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 15 10:45:30.035513 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.034454    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 15 10:45:30.035513 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:30.034489    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 15 10:45:30.036895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.036873    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 10:45:30.049844 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.049830    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:45:30.050793 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.050769    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:45:30.050870 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.050806    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:45:30.050870 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.050823    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:45:30.050870 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.050847    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-26.ec2.internal"
Jul 15 10:45:30.061647 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.061611    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-26.ec2.internal"
Jul 15 10:45:30.061647 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:30.061631    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-26.ec2.internal\": node \"ip-10-0-133-26.ec2.internal\" not found"
Jul 15 10:45:30.081931 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:30.081907    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Jul 15 10:45:30.135563 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.135543    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"]
Jul 15 10:45:30.135640 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.135628    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:45:30.138265 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.138246    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:45:30.138330 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.138277    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:45:30.138330 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.138287    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:45:30.139499 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.139488    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:45:30.139640 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.139626    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Jul 15 10:45:30.139678 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.139661    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:45:30.140209 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.140196    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:45:30.140272 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.140220    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:45:30.140272 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.140229    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:45:30.140723 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.140706    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:45:30.140780 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.140735    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:45:30.140780 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.140745    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:45:30.141824 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.141810    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Jul 15 10:45:30.141865 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.141838    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 15 10:45:30.143144 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.143130    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientMemory"
Jul 15 10:45:30.143205 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.143162    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasNoDiskPressure"
Jul 15 10:45:30.143205 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.143176    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeHasSufficientPID"
Jul 15 10:45:30.159105 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:30.159086    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-26.ec2.internal\" not found" node="ip-10-0-133-26.ec2.internal"
Jul 15 10:45:30.162502 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:30.162482    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-26.ec2.internal\" not found" node="ip-10-0-133-26.ec2.internal"
Jul 15 10:45:30.182256 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:30.182234    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Jul 15 10:45:30.193083 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.193064    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/395ae80f9c797fbcfa996f0e1f6c0bf1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"395ae80f9c797fbcfa996f0e1f6c0bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Jul 15 10:45:30.193166 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.193092    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/395ae80f9c797fbcfa996f0e1f6c0bf1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"395ae80f9c797fbcfa996f0e1f6c0bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Jul 15 10:45:30.193166 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.193109    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/87afa8806d81c1e28dbf5cedd9e00a8e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-26.ec2.internal\" (UID: \"87afa8806d81c1e28dbf5cedd9e00a8e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Jul 15 10:45:30.282865 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:30.282840    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Jul 15 10:45:30.294190 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.294171    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/395ae80f9c797fbcfa996f0e1f6c0bf1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"395ae80f9c797fbcfa996f0e1f6c0bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Jul 15 10:45:30.294265 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.294201    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/395ae80f9c797fbcfa996f0e1f6c0bf1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"395ae80f9c797fbcfa996f0e1f6c0bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Jul 15 10:45:30.294265 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.294217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/87afa8806d81c1e28dbf5cedd9e00a8e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-26.ec2.internal\" (UID: \"87afa8806d81c1e28dbf5cedd9e00a8e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Jul 15 10:45:30.294265 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.294264    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/87afa8806d81c1e28dbf5cedd9e00a8e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-26.ec2.internal\" (UID: \"87afa8806d81c1e28dbf5cedd9e00a8e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Jul 15 10:45:30.294373 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.294266    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/395ae80f9c797fbcfa996f0e1f6c0bf1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"395ae80f9c797fbcfa996f0e1f6c0bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Jul 15 10:45:30.294373 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.294307    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/395ae80f9c797fbcfa996f0e1f6c0bf1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal\" (UID: \"395ae80f9c797fbcfa996f0e1f6c0bf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Jul 15 10:45:30.383973 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:30.383874    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Jul 15 10:45:30.461414 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.461379    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Jul 15 10:45:30.466146 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:30.466122    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Jul 15 10:45:30.484003 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:30.483975    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Jul 15 10:45:30.584505 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:30.584454    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Jul 15 10:45:30.685062 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:30.684979    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Jul 15 10:45:32.012424 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:32.012377    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Jul 15 10:45:32.057845 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:32.057820    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 15 10:45:32.058416 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:32.057978    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 15 10:45:32.112484 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:32.112458    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Jul 15 10:45:32.142727 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:32.142688    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 10:40:31 +0000 UTC" deadline="2028-03-04 07:09:36.093106894 +0000 UTC"
Jul 15 10:45:32.142727 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:32.142722    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14348h24m3.950389512s"
Jul 15 10:45:32.212952 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:32.212923    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Jul 15 10:45:32.296784 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:32.296745    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395ae80f9c797fbcfa996f0e1f6c0bf1.slice/crio-181a6c4ba1d57861eefa8049ed331701ea0d9d3d4313e1a3e515c3456dae9f72 WatchSource:0}: Error finding container 181a6c4ba1d57861eefa8049ed331701ea0d9d3d4313e1a3e515c3456dae9f72: Status 404 returned error can't find the container with id 181a6c4ba1d57861eefa8049ed331701ea0d9d3d4313e1a3e515c3456dae9f72
Jul 15 10:45:32.297074 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:32.297054    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87afa8806d81c1e28dbf5cedd9e00a8e.slice/crio-fa4198e17f5d05be52eb426f1716df01beebde139caf307cb038b1ac9964c8ed WatchSource:0}: Error finding container fa4198e17f5d05be52eb426f1716df01beebde139caf307cb038b1ac9964c8ed: Status 404 returned error can't find the container with id fa4198e17f5d05be52eb426f1716df01beebde139caf307cb038b1ac9964c8ed
Jul 15 10:45:32.300457 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:32.300443    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 10:45:32.313554 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:32.313535    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-26.ec2.internal\" not found"
Jul 15 10:45:32.358667 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:32.358645    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 10:45:32.417308 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:32.417285    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal"
Jul 15 10:45:32.431725 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:32.431706    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 15 10:45:32.433162 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:32.433150    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal"
Jul 15 10:45:32.445098 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:32.445078    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 15 10:45:32.479239 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:32.479177    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 10:45:32.496054 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:32.496024    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 10:45:33.034855 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.034822    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 15 10:45:33.099509 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.099261    2566 apiserver.go:52] "Watching apiserver"
Jul 15 10:45:33.107529 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.107477    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 15 10:45:33.107876 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.107851    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-2jprt","kube-system/konnectivity-agent-wcspb","kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal","openshift-image-registry/node-ca-hxflc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal","openshift-multus/multus-additional-cni-plugins-4qs25","openshift-ovn-kubernetes/ovnkube-node-5tqtw","kube-system/global-pull-secret-syncer-5ffl8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7","openshift-cluster-node-tuning-operator/tuned-68n9x","openshift-dns/node-resolver-ww56m","openshift-multus/multus-xtlgc","openshift-multus/network-metrics-daemon-hkvw8","openshift-network-diagnostics/network-check-target-m7s5x"]
Jul 15 10:45:33.109584 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.109564    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.110735 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.110709    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ww56m"
Jul 15 10:45:33.112084 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.112064    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:33.112175 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.112142    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:45:33.113976 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.113488    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hxflc"
Jul 15 10:45:33.114469 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.114453    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:33.114560 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.114536    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:45:33.116512 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.116474    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.117344 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.117328    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 15 10:45:33.118118 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.118052    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.120985 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.120971    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:33.121089 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.121015    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 15 10:45:33.121089 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.121034    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkvw8" podUID="8a1c4669-2697-411f-80ff-c3c5819361dc"
Jul 15 10:45:33.121463 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.121443    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 15 10:45:33.121925 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.121906    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 15 10:45:33.122040 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.122021    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wcspb"
Jul 15 10:45:33.122101 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.122044    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-njdkb\""
Jul 15 10:45:33.122242 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.122225    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 15 10:45:33.122427 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.122387    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 15 10:45:33.122619 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.122594    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 15 10:45:33.122847 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.122824    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dgdqb\""
Jul 15 10:45:33.123432 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.123411    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.124015 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.123997    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 15 10:45:33.124295 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.124271    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 15 10:45:33.124571 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.124552    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 15 10:45:33.124662 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.124650    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 15 10:45:33.125002 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.124963    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 15 10:45:33.125091 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.124968    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2jprt"
Jul 15 10:45:33.126696 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.126065    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 15 10:45:33.126696 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.125694    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.127136    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.128231    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mr2gx\""
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.128601    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.128729    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.128960    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.128730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-socket-dir\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129129    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqzfc\" (UniqueName: \"kubernetes.io/projected/63ae8640-95c8-4f02-a7a9-df431281ef52-kube-api-access-gqzfc\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129169    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dkkn2\""
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129166    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-var-lib-openvswitch\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129224    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20239449-cde8-4cb1-9c0f-dad9abb7cd58-env-overrides\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129243    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20239449-cde8-4cb1-9c0f-dad9abb7cd58-ovnkube-script-lib\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129264    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kz6w\" (UniqueName: \"kubernetes.io/projected/6918f75f-21cc-47c8-9332-c1055bdbf5df-kube-api-access-9kz6w\") pod \"node-resolver-ww56m\" (UID: \"6918f75f-21cc-47c8-9332-c1055bdbf5df\") " pod="openshift-dns/node-resolver-ww56m"
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6321df80-acdb-482c-bb4e-d04645f9131f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129304    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6321df80-acdb-482c-bb4e-d04645f9131f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129370    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129402    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-sys-fs\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129484    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-kubelet\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129534    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-run-systemd\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.129895 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129566    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-etc-openvswitch\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-cni-bin\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129636    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znxrh\" (UniqueName: \"kubernetes.io/projected/20239449-cde8-4cb1-9c0f-dad9abb7cd58-kube-api-access-znxrh\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129667    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6321df80-acdb-482c-bb4e-d04645f9131f-cnibin\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129705    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129725    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-etc-selinux\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129735    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tq4mh\""
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129755    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-slash\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-run-ovn-kubernetes\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129813    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-cni-netd\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129844    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129788    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.129914    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.130426    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6321df80-acdb-482c-bb4e-d04645f9131f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.130467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c118fdb2-4d52-4883-99c6-cf8609b6d8dd-host\") pod \"node-ca-hxflc\" (UID: \"c118fdb2-4d52-4883-99c6-cf8609b6d8dd\") " pod="openshift-image-registry/node-ca-hxflc"
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.130516    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rgnz\" (UniqueName: \"kubernetes.io/projected/c118fdb2-4d52-4883-99c6-cf8609b6d8dd-kube-api-access-5rgnz\") pod \"node-ca-hxflc\" (UID: \"c118fdb2-4d52-4883-99c6-cf8609b6d8dd\") " pod="openshift-image-registry/node-ca-hxflc"
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.130546    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpv7m\" (UniqueName: \"kubernetes.io/projected/6321df80-acdb-482c-bb4e-d04645f9131f-kube-api-access-qpv7m\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.130686    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.130838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.130834    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c118fdb2-4d52-4883-99c6-cf8609b6d8dd-serviceca\") pod \"node-ca-hxflc\" (UID: \"c118fdb2-4d52-4883-99c6-cf8609b6d8dd\") " pod="openshift-image-registry/node-ca-hxflc"
Jul 15 10:45:33.131648 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.130867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-dbus\") pod \"global-pull-secret-syncer-5ffl8\" (UID: \"dae7d2b5-9b3e-4ffe-85a3-087ad2313052\") " pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:33.131648 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.130898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-run-openvswitch\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.131648 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.130928    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-node-log\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.131648 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.131012    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20239449-cde8-4cb1-9c0f-dad9abb7cd58-ovnkube-config\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.131648 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.131047    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6321df80-acdb-482c-bb4e-d04645f9131f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.131648 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.131085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjtrg\" (UniqueName: \"kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg\") pod \"network-check-target-m7s5x\" (UID: \"11345472-e392-4a24-b60c-43b5fe6dae64\") " pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:33.131648 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.131116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-kubelet-config\") pod \"global-pull-secret-syncer-5ffl8\" (UID: \"dae7d2b5-9b3e-4ffe-85a3-087ad2313052\") " pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:33.131648 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.131147    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-run-ovn\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.131648 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.131175    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-log-socket\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.131648 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.131207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20239449-cde8-4cb1-9c0f-dad9abb7cd58-ovn-node-metrics-cert\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.131648 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.131313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6918f75f-21cc-47c8-9332-c1055bdbf5df-hosts-file\") pod \"node-resolver-ww56m\" (UID: \"6918f75f-21cc-47c8-9332-c1055bdbf5df\") " pod="openshift-dns/node-resolver-ww56m"
Jul 15 10:45:33.131648 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.131365    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6321df80-acdb-482c-bb4e-d04645f9131f-os-release\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.131648 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.131397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-registration-dir\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.131648 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.131427    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-run-netns\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.132255 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.131915    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6918f75f-21cc-47c8-9332-c1055bdbf5df-tmp-dir\") pod \"node-resolver-ww56m\" (UID: \"6918f75f-21cc-47c8-9332-c1055bdbf5df\") " pod="openshift-dns/node-resolver-ww56m"
Jul 15 10:45:33.132255 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.131966    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6321df80-acdb-482c-bb4e-d04645f9131f-system-cni-dir\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.132255 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.131994    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret\") pod \"global-pull-secret-syncer-5ffl8\" (UID: \"dae7d2b5-9b3e-4ffe-85a3-087ad2313052\") " pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:33.132255 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.132026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-device-dir\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.132255 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.132054    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-systemd-units\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.132849 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.132825    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gf6sq\""
Jul 15 10:45:33.132965 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.132952    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qlf7m\""
Jul 15 10:45:33.133310 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.133287    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 15 10:45:33.133379 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.133314    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 15 10:45:33.133558 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.133533    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 15 10:45:33.133873 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.133853    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 15 10:45:33.134302 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.134282    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 15 10:45:33.134401 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.134355    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dph4m\""
Jul 15 10:45:33.138961 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.138943    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 15 10:45:33.139701 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.139680    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 15 10:45:33.140703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.140683    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7mrhv\""
Jul 15 10:45:33.147205 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.147167    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 15 10:45:33.185882 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.185832    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t8kf9"
Jul 15 10:45:33.193362 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.193340    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t8kf9"
Jul 15 10:45:33.218848 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.218820    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 15 10:45:33.232707 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.232682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-run-openvswitch\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.232824 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.232714    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-systemd\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.232824 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.232743    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-sys\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.232824 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.232765    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-multus-cni-dir\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.232824 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.232787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-multus-socket-dir-parent\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.232824 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.232811    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-var-lib-cni-bin\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.233014 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.232790    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-run-openvswitch\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.233014 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.232836    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-run-multus-certs\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.233014 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.232870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-log-socket\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.233014 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.232901    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6918f75f-21cc-47c8-9332-c1055bdbf5df-hosts-file\") pod \"node-resolver-ww56m\" (UID: \"6918f75f-21cc-47c8-9332-c1055bdbf5df\") " pod="openshift-dns/node-resolver-ww56m"
Jul 15 10:45:33.233014 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.232921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-var-lib-kubelet\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.233014 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.232945    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-cni-binary-copy\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.233014 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.232964    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-log-socket\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.233014 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.232969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-run-netns\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.233014 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-run-netns\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.233014 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233014    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret\") pod \"global-pull-secret-syncer-5ffl8\" (UID: \"dae7d2b5-9b3e-4ffe-85a3-087ad2313052\") " pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:33.233556 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b240a0c-d42c-4dba-b5ea-9be576447190-host-slash\") pod \"iptables-alerter-2jprt\" (UID: \"4b240a0c-d42c-4dba-b5ea-9be576447190\") " pod="openshift-network-operator/iptables-alerter-2jprt"
Jul 15 10:45:33.233556 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233063    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6918f75f-21cc-47c8-9332-c1055bdbf5df-hosts-file\") pod \"node-resolver-ww56m\" (UID: \"6918f75f-21cc-47c8-9332-c1055bdbf5df\") " pod="openshift-dns/node-resolver-ww56m"
Jul 15 10:45:33.233556 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233081    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-var-lib-kubelet\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.233556 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-device-dir\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.233556 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-etc-kubernetes\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.233556 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233163    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-socket-dir\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.233556 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.233183    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:33.233556 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233187    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqzfc\" (UniqueName: \"kubernetes.io/projected/63ae8640-95c8-4f02-a7a9-df431281ef52-kube-api-access-gqzfc\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.233556 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233210    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20239449-cde8-4cb1-9c0f-dad9abb7cd58-env-overrides\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.233556 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.233254    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret podName:dae7d2b5-9b3e-4ffe-85a3-087ad2313052 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:33.733222 +0000 UTC m=+2.963189522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret") pod "global-pull-secret-syncer-5ffl8" (UID: "dae7d2b5-9b3e-4ffe-85a3-087ad2313052") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:33.233556 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233270    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20239449-cde8-4cb1-9c0f-dad9abb7cd58-ovnkube-script-lib\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.233556 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kz6w\" (UniqueName: \"kubernetes.io/projected/6918f75f-21cc-47c8-9332-c1055bdbf5df-kube-api-access-9kz6w\") pod \"node-resolver-ww56m\" (UID: \"6918f75f-21cc-47c8-9332-c1055bdbf5df\") " pod="openshift-dns/node-resolver-ww56m"
Jul 15 10:45:33.233556 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233317    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6321df80-acdb-482c-bb4e-d04645f9131f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.233556 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-sysconfig\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.233556 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-sysctl-d\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.233556 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233405    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-sys-fs\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.233556 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233434    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-cni-bin\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.234359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233456    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6321df80-acdb-482c-bb4e-d04645f9131f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.234359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-kubelet-config\") pod \"global-pull-secret-syncer-5ffl8\" (UID: \"dae7d2b5-9b3e-4ffe-85a3-087ad2313052\") " pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:33.234359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-run\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.234359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233554    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-system-cni-dir\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.234359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233576    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-os-release\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.234359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233598    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-var-lib-cni-multus\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.234359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-run-ovn-kubernetes\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.234359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233647    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cdfb31e6-b8ba-4726-afb8-6892dd4adcc7-agent-certs\") pod \"konnectivity-agent-wcspb\" (UID: \"cdfb31e6-b8ba-4726-afb8-6892dd4adcc7\") " pod="kube-system/konnectivity-agent-wcspb"
Jul 15 10:45:33.234359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233671    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-kubernetes\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.234359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233693    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-multus-conf-dir\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.234359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233713    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20239449-cde8-4cb1-9c0f-dad9abb7cd58-env-overrides\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.234359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-multus-daemon-config\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.234359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233747    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxppc\" (UniqueName: \"kubernetes.io/projected/4b240a0c-d42c-4dba-b5ea-9be576447190-kube-api-access-lxppc\") pod \"iptables-alerter-2jprt\" (UID: \"4b240a0c-d42c-4dba-b5ea-9be576447190\") " pod="openshift-network-operator/iptables-alerter-2jprt"
Jul 15 10:45:33.234359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-node-log\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.234359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233810    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20239449-cde8-4cb1-9c0f-dad9abb7cd58-ovnkube-config\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.234359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233822    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-device-dir\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.234359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233836    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6321df80-acdb-482c-bb4e-d04645f9131f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.235129 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjtrg\" (UniqueName: \"kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg\") pod \"network-check-target-m7s5x\" (UID: \"11345472-e392-4a24-b60c-43b5fe6dae64\") " pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:33.235129 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233883    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78b844b5-2671-4bfa-b242-40537d33c32f-etc-tuned\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.235129 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233905    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78b844b5-2671-4bfa-b242-40537d33c32f-tmp\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.235129 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233935    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vfgm\" (UniqueName: \"kubernetes.io/projected/78b844b5-2671-4bfa-b242-40537d33c32f-kube-api-access-6vfgm\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.235129 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233957    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-run-ovn\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.235129 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233978    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20239449-cde8-4cb1-9c0f-dad9abb7cd58-ovn-node-metrics-cert\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.235129 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6321df80-acdb-482c-bb4e-d04645f9131f-os-release\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.235129 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234023    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs\") pod \"network-metrics-daemon-hkvw8\" (UID: \"8a1c4669-2697-411f-80ff-c3c5819361dc\") " pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:33.235129 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234046    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-cnibin\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.235129 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234071    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-hostroot\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.235129 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234094    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz2j8\" (UniqueName: \"kubernetes.io/projected/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-kube-api-access-sz2j8\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.235129 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-registration-dir\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.235129 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6918f75f-21cc-47c8-9332-c1055bdbf5df-tmp-dir\") pod \"node-resolver-ww56m\" (UID: \"6918f75f-21cc-47c8-9332-c1055bdbf5df\") " pod="openshift-dns/node-resolver-ww56m"
Jul 15 10:45:33.235129 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234181    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6321df80-acdb-482c-bb4e-d04645f9131f-system-cni-dir\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.235129 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-lib-modules\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.235129 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234227    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-host\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.235129 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234268    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-run-netns\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.235920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-systemd-units\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.235920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234322    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cdfb31e6-b8ba-4726-afb8-6892dd4adcc7-konnectivity-ca\") pod \"konnectivity-agent-wcspb\" (UID: \"cdfb31e6-b8ba-4726-afb8-6892dd4adcc7\") " pod="kube-system/konnectivity-agent-wcspb"
Jul 15 10:45:33.235920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234348    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-modprobe-d\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.235920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234372    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-var-lib-openvswitch\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.235920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6321df80-acdb-482c-bb4e-d04645f9131f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.235920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234424    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpv7m\" (UniqueName: \"kubernetes.io/projected/6321df80-acdb-482c-bb4e-d04645f9131f-kube-api-access-qpv7m\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.235920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234533    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-node-log\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.235920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234831    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-sys-fs\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.235920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234882    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-cni-bin\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.235920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20239449-cde8-4cb1-9c0f-dad9abb7cd58-ovnkube-config\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.235920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.234994    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6321df80-acdb-482c-bb4e-d04645f9131f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.235920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235032    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6321df80-acdb-482c-bb4e-d04645f9131f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.235920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235089    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-kubelet-config\") pod \"global-pull-secret-syncer-5ffl8\" (UID: \"dae7d2b5-9b3e-4ffe-85a3-087ad2313052\") " pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:33.235920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235096    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-run-ovn-kubernetes\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.235920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235174    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.235920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235208    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-kubelet\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.235920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-run-systemd\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.236703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235259    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-etc-openvswitch\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.236703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235283    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znxrh\" (UniqueName: \"kubernetes.io/projected/20239449-cde8-4cb1-9c0f-dad9abb7cd58-kube-api-access-znxrh\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.236703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235307    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6321df80-acdb-482c-bb4e-d04645f9131f-cnibin\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.236703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c118fdb2-4d52-4883-99c6-cf8609b6d8dd-host\") pod \"node-ca-hxflc\" (UID: \"c118fdb2-4d52-4883-99c6-cf8609b6d8dd\") " pod="openshift-image-registry/node-ca-hxflc"
Jul 15 10:45:33.236703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235360    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rgnz\" (UniqueName: \"kubernetes.io/projected/c118fdb2-4d52-4883-99c6-cf8609b6d8dd-kube-api-access-5rgnz\") pod \"node-ca-hxflc\" (UID: \"c118fdb2-4d52-4883-99c6-cf8609b6d8dd\") " pod="openshift-image-registry/node-ca-hxflc"
Jul 15 10:45:33.236703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235415    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-etc-selinux\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.236703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235441    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-slash\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.236703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235465    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-cni-netd\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.236703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235485    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6321df80-acdb-482c-bb4e-d04645f9131f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.236703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235511    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.236703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-dbus\") pod \"global-pull-secret-syncer-5ffl8\" (UID: \"dae7d2b5-9b3e-4ffe-85a3-087ad2313052\") " pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:33.236703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235568    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.236703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235576    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8lsz\" (UniqueName: \"kubernetes.io/projected/8a1c4669-2697-411f-80ff-c3c5819361dc-kube-api-access-r8lsz\") pod \"network-metrics-daemon-hkvw8\" (UID: \"8a1c4669-2697-411f-80ff-c3c5819361dc\") " pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:33.236703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-sysctl-conf\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.236703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235628    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b240a0c-d42c-4dba-b5ea-9be576447190-iptables-alerter-script\") pod \"iptables-alerter-2jprt\" (UID: \"4b240a0c-d42c-4dba-b5ea-9be576447190\") " pod="openshift-network-operator/iptables-alerter-2jprt"
Jul 15 10:45:33.236703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235633    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-run-systemd\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.236703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235653    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c118fdb2-4d52-4883-99c6-cf8609b6d8dd-serviceca\") pod \"node-ca-hxflc\" (UID: \"c118fdb2-4d52-4883-99c6-cf8609b6d8dd\") " pod="openshift-image-registry/node-ca-hxflc"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235669    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-etc-openvswitch\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235678    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-run-k8s-cni-cncf-io\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.233910    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-socket-dir\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-run-ovn\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235938    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-systemd-units\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235988    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6321df80-acdb-482c-bb4e-d04645f9131f-os-release\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.236027    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-etc-selinux\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.236079    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-slash\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.236090    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63ae8640-95c8-4f02-a7a9-df431281ef52-registration-dir\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.236123    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-cni-netd\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.236182    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c118fdb2-4d52-4883-99c6-cf8609b6d8dd-host\") pod \"node-ca-hxflc\" (UID: \"c118fdb2-4d52-4883-99c6-cf8609b6d8dd\") " pod="openshift-image-registry/node-ca-hxflc"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.236217    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.236294    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-var-lib-openvswitch\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.236345    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6918f75f-21cc-47c8-9332-c1055bdbf5df-tmp-dir\") pod \"node-resolver-ww56m\" (UID: \"6918f75f-21cc-47c8-9332-c1055bdbf5df\") " pod="openshift-dns/node-resolver-ww56m"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.236730    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6321df80-acdb-482c-bb4e-d04645f9131f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.236924    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.237051    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-dbus\") pod \"global-pull-secret-syncer-5ffl8\" (UID: \"dae7d2b5-9b3e-4ffe-85a3-087ad2313052\") " pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:33.237545 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.237415    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c118fdb2-4d52-4883-99c6-cf8609b6d8dd-serviceca\") pod \"node-ca-hxflc\" (UID: \"c118fdb2-4d52-4883-99c6-cf8609b6d8dd\") " pod="openshift-image-registry/node-ca-hxflc"
Jul 15 10:45:33.238409 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235606    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20239449-cde8-4cb1-9c0f-dad9abb7cd58-host-kubelet\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.238409 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.235843    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6321df80-acdb-482c-bb4e-d04645f9131f-cnibin\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.238409 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.236239    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6321df80-acdb-482c-bb4e-d04645f9131f-system-cni-dir\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.238409 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.237473    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20239449-cde8-4cb1-9c0f-dad9abb7cd58-ovnkube-script-lib\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.239912 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.239688    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20239449-cde8-4cb1-9c0f-dad9abb7cd58-ovn-node-metrics-cert\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.246985 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.246957    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqzfc\" (UniqueName: \"kubernetes.io/projected/63ae8640-95c8-4f02-a7a9-df431281ef52-kube-api-access-gqzfc\") pod \"aws-ebs-csi-driver-node-5pkg7\" (UID: \"63ae8640-95c8-4f02-a7a9-df431281ef52\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.246985 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.246978    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znxrh\" (UniqueName: \"kubernetes.io/projected/20239449-cde8-4cb1-9c0f-dad9abb7cd58-kube-api-access-znxrh\") pod \"ovnkube-node-5tqtw\" (UID: \"20239449-cde8-4cb1-9c0f-dad9abb7cd58\") " pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.249772 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.249745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rgnz\" (UniqueName: \"kubernetes.io/projected/c118fdb2-4d52-4883-99c6-cf8609b6d8dd-kube-api-access-5rgnz\") pod \"node-ca-hxflc\" (UID: \"c118fdb2-4d52-4883-99c6-cf8609b6d8dd\") " pod="openshift-image-registry/node-ca-hxflc"
Jul 15 10:45:33.250095 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.250078    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpv7m\" (UniqueName: \"kubernetes.io/projected/6321df80-acdb-482c-bb4e-d04645f9131f-kube-api-access-qpv7m\") pod \"multus-additional-cni-plugins-4qs25\" (UID: \"6321df80-acdb-482c-bb4e-d04645f9131f\") " pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.250320 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.250304    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:45:33.250320 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.250326    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:45:33.250453 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.250339    2566 projected.go:196] Error preparing data for projected volume kube-api-access-tjtrg for pod openshift-network-diagnostics/network-check-target-m7s5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:33.250453 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.250393    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg podName:11345472-e392-4a24-b60c-43b5fe6dae64 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:33.750377109 +0000 UTC m=+2.980344621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tjtrg" (UniqueName: "kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg") pod "network-check-target-m7s5x" (UID: "11345472-e392-4a24-b60c-43b5fe6dae64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:33.251159 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.251137    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kz6w\" (UniqueName: \"kubernetes.io/projected/6918f75f-21cc-47c8-9332-c1055bdbf5df-kube-api-access-9kz6w\") pod \"node-resolver-ww56m\" (UID: \"6918f75f-21cc-47c8-9332-c1055bdbf5df\") " pod="openshift-dns/node-resolver-ww56m"
Jul 15 10:45:33.268132 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.268061    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal" event={"ID":"395ae80f9c797fbcfa996f0e1f6c0bf1","Type":"ContainerStarted","Data":"181a6c4ba1d57861eefa8049ed331701ea0d9d3d4313e1a3e515c3456dae9f72"}
Jul 15 10:45:33.269680 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.269654    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal" event={"ID":"87afa8806d81c1e28dbf5cedd9e00a8e","Type":"ContainerStarted","Data":"fa4198e17f5d05be52eb426f1716df01beebde139caf307cb038b1ac9964c8ed"}
Jul 15 10:45:33.336140 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336056    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78b844b5-2671-4bfa-b242-40537d33c32f-etc-tuned\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.336140 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78b844b5-2671-4bfa-b242-40537d33c32f-tmp\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.336140 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vfgm\" (UniqueName: \"kubernetes.io/projected/78b844b5-2671-4bfa-b242-40537d33c32f-kube-api-access-6vfgm\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.336404 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336236    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs\") pod \"network-metrics-daemon-hkvw8\" (UID: \"8a1c4669-2697-411f-80ff-c3c5819361dc\") " pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:33.336404 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-cnibin\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.336404 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336300    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-hostroot\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.336404 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz2j8\" (UniqueName: \"kubernetes.io/projected/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-kube-api-access-sz2j8\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.336404 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336351    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-lib-modules\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.336404 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336374    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-host\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.336404 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.336391    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:33.336748 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336426    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-cnibin\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.336748 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336442    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-run-netns\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.336748 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.336462    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs podName:8a1c4669-2697-411f-80ff-c3c5819361dc nodeName:}" failed. No retries permitted until 2026-07-15 10:45:33.836438665 +0000 UTC m=+3.066406188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs") pod "network-metrics-daemon-hkvw8" (UID: "8a1c4669-2697-411f-80ff-c3c5819361dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:33.336748 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-run-netns\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.336748 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cdfb31e6-b8ba-4726-afb8-6892dd4adcc7-konnectivity-ca\") pod \"konnectivity-agent-wcspb\" (UID: \"cdfb31e6-b8ba-4726-afb8-6892dd4adcc7\") " pod="kube-system/konnectivity-agent-wcspb"
Jul 15 10:45:33.336748 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336555    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-modprobe-d\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.336748 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336588    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-lib-modules\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.336748 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336596    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8lsz\" (UniqueName: \"kubernetes.io/projected/8a1c4669-2697-411f-80ff-c3c5819361dc-kube-api-access-r8lsz\") pod \"network-metrics-daemon-hkvw8\" (UID: \"8a1c4669-2697-411f-80ff-c3c5819361dc\") " pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:33.336748 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336632    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-sysctl-conf\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.336748 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336662    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b240a0c-d42c-4dba-b5ea-9be576447190-iptables-alerter-script\") pod \"iptables-alerter-2jprt\" (UID: \"4b240a0c-d42c-4dba-b5ea-9be576447190\") " pod="openshift-network-operator/iptables-alerter-2jprt"
Jul 15 10:45:33.336748 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-run-k8s-cni-cncf-io\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.336748 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-systemd\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.336748 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336712    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-host\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.336748 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336735    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-hostroot\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.336748 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336738    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-sys\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-multus-cni-dir\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336782    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-sys\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336798    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-multus-socket-dir-parent\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-var-lib-cni-bin\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-run-multus-certs\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336865    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-var-lib-kubelet\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336879    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-sysctl-conf\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-cni-binary-copy\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336926    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b240a0c-d42c-4dba-b5ea-9be576447190-host-slash\") pod \"iptables-alerter-2jprt\" (UID: \"4b240a0c-d42c-4dba-b5ea-9be576447190\") " pod="openshift-network-operator/iptables-alerter-2jprt"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336948    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-var-lib-kubelet\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-modprobe-d\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.336974    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-etc-kubernetes\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-sysconfig\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337007    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-run-k8s-cni-cncf-io\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-sysctl-d\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337064    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-sysconfig\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337073    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-run\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.337514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337096    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-system-cni-dir\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337114    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-multus-cni-dir\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-os-release\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337161    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-multus-socket-dir-parent\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337209    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-sysctl-d\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337245    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-run\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337262    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-os-release\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337280    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cdfb31e6-b8ba-4726-afb8-6892dd4adcc7-konnectivity-ca\") pod \"konnectivity-agent-wcspb\" (UID: \"cdfb31e6-b8ba-4726-afb8-6892dd4adcc7\") " pod="kube-system/konnectivity-agent-wcspb"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-system-cni-dir\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337305    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-var-lib-cni-bin\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337331    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b240a0c-d42c-4dba-b5ea-9be576447190-iptables-alerter-script\") pod \"iptables-alerter-2jprt\" (UID: \"4b240a0c-d42c-4dba-b5ea-9be576447190\") " pod="openshift-network-operator/iptables-alerter-2jprt"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337355    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b240a0c-d42c-4dba-b5ea-9be576447190-host-slash\") pod \"iptables-alerter-2jprt\" (UID: \"4b240a0c-d42c-4dba-b5ea-9be576447190\") " pod="openshift-network-operator/iptables-alerter-2jprt"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337359    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-systemd\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337384    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-run-multus-certs\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337390    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-var-lib-cni-multus\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337392    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-var-lib-kubelet\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337416    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cdfb31e6-b8ba-4726-afb8-6892dd4adcc7-agent-certs\") pod \"konnectivity-agent-wcspb\" (UID: \"cdfb31e6-b8ba-4726-afb8-6892dd4adcc7\") " pod="kube-system/konnectivity-agent-wcspb"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337431    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-var-lib-kubelet\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.338196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337440    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-kubernetes\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.338854 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337452    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-host-var-lib-cni-multus\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.338854 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337515    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-etc-kubernetes\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.338854 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337547    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78b844b5-2671-4bfa-b242-40537d33c32f-etc-kubernetes\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.338854 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337555    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-multus-conf-dir\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.338854 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337591    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-multus-conf-dir\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.338854 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337625    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-multus-daemon-config\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.338854 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337664    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxppc\" (UniqueName: \"kubernetes.io/projected/4b240a0c-d42c-4dba-b5ea-9be576447190-kube-api-access-lxppc\") pod \"iptables-alerter-2jprt\" (UID: \"4b240a0c-d42c-4dba-b5ea-9be576447190\") " pod="openshift-network-operator/iptables-alerter-2jprt"
Jul 15 10:45:33.338854 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.337848    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-cni-binary-copy\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.338854 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.338257    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-multus-daemon-config\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.339216 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.339190    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78b844b5-2671-4bfa-b242-40537d33c32f-etc-tuned\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.339600 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.339571    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78b844b5-2671-4bfa-b242-40537d33c32f-tmp\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.339915 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.339893    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cdfb31e6-b8ba-4726-afb8-6892dd4adcc7-agent-certs\") pod \"konnectivity-agent-wcspb\" (UID: \"cdfb31e6-b8ba-4726-afb8-6892dd4adcc7\") " pod="kube-system/konnectivity-agent-wcspb"
Jul 15 10:45:33.351774 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.351753    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz2j8\" (UniqueName: \"kubernetes.io/projected/6af7557c-6bfb-4ce6-a879-f95ca89cabdf-kube-api-access-sz2j8\") pod \"multus-xtlgc\" (UID: \"6af7557c-6bfb-4ce6-a879-f95ca89cabdf\") " pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.355980 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.355959    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8lsz\" (UniqueName: \"kubernetes.io/projected/8a1c4669-2697-411f-80ff-c3c5819361dc-kube-api-access-r8lsz\") pod \"network-metrics-daemon-hkvw8\" (UID: \"8a1c4669-2697-411f-80ff-c3c5819361dc\") " pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:33.357918 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.357896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vfgm\" (UniqueName: \"kubernetes.io/projected/78b844b5-2671-4bfa-b242-40537d33c32f-kube-api-access-6vfgm\") pod \"tuned-68n9x\" (UID: \"78b844b5-2671-4bfa-b242-40537d33c32f\") " pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.358194 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.358176    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxppc\" (UniqueName: \"kubernetes.io/projected/4b240a0c-d42c-4dba-b5ea-9be576447190-kube-api-access-lxppc\") pod \"iptables-alerter-2jprt\" (UID: \"4b240a0c-d42c-4dba-b5ea-9be576447190\") " pod="openshift-network-operator/iptables-alerter-2jprt"
Jul 15 10:45:33.423069 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.423036    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:33.430787 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.430764    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ww56m"
Jul 15 10:45:33.443507 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.443473    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hxflc"
Jul 15 10:45:33.448142 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.448110    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4qs25"
Jul 15 10:45:33.455219 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.455203    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7"
Jul 15 10:45:33.463238 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.463217    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wcspb"
Jul 15 10:45:33.470979 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.470959    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-68n9x"
Jul 15 10:45:33.480651 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.480625    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2jprt"
Jul 15 10:45:33.487415 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.487396    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xtlgc"
Jul 15 10:45:33.740865 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.740763    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret\") pod \"global-pull-secret-syncer-5ffl8\" (UID: \"dae7d2b5-9b3e-4ffe-85a3-087ad2313052\") " pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:33.741026 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.740885    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:33.741026 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.740950    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret podName:dae7d2b5-9b3e-4ffe-85a3-087ad2313052 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:34.740929464 +0000 UTC m=+3.970896977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret") pod "global-pull-secret-syncer-5ffl8" (UID: "dae7d2b5-9b3e-4ffe-85a3-087ad2313052") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:33.842093 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.842059    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjtrg\" (UniqueName: \"kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg\") pod \"network-check-target-m7s5x\" (UID: \"11345472-e392-4a24-b60c-43b5fe6dae64\") " pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:33.842093 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:33.842094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs\") pod \"network-metrics-daemon-hkvw8\" (UID: \"8a1c4669-2697-411f-80ff-c3c5819361dc\") " pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:33.842317 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.842216    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:33.842317 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.842257    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:45:33.842317 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.842273    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs podName:8a1c4669-2697-411f-80ff-c3c5819361dc nodeName:}" failed. No retries permitted until 2026-07-15 10:45:34.842254363 +0000 UTC m=+4.072221872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs") pod "network-metrics-daemon-hkvw8" (UID: "8a1c4669-2697-411f-80ff-c3c5819361dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:33.842317 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.842277    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:45:33.842317 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.842294    2566 projected.go:196] Error preparing data for projected volume kube-api-access-tjtrg for pod openshift-network-diagnostics/network-check-target-m7s5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:33.842540 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:33.842347    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg podName:11345472-e392-4a24-b60c-43b5fe6dae64 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:34.842331471 +0000 UTC m=+4.072298994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tjtrg" (UniqueName: "kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg") pod "network-check-target-m7s5x" (UID: "11345472-e392-4a24-b60c-43b5fe6dae64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:34.133086 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:34.133055    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6321df80_acdb_482c_bb4e_d04645f9131f.slice/crio-bbc6e493e81caf227fdd35d36242de369fff225f38fd97151998ae67d321a049 WatchSource:0}: Error finding container bbc6e493e81caf227fdd35d36242de369fff225f38fd97151998ae67d321a049: Status 404 returned error can't find the container with id bbc6e493e81caf227fdd35d36242de369fff225f38fd97151998ae67d321a049
Jul 15 10:45:34.133949 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:34.133927    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b240a0c_d42c_4dba_b5ea_9be576447190.slice/crio-817e5be38617128713f6660e925fcba4cf794a7147631ae4b632d80aaf9b35b9 WatchSource:0}: Error finding container 817e5be38617128713f6660e925fcba4cf794a7147631ae4b632d80aaf9b35b9: Status 404 returned error can't find the container with id 817e5be38617128713f6660e925fcba4cf794a7147631ae4b632d80aaf9b35b9
Jul 15 10:45:34.134998 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:34.134976    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ae8640_95c8_4f02_a7a9_df431281ef52.slice/crio-fe68e76ea504f811cd5a93e6d719c6958a0451bd89ecec7ed32982fad4a98a71 WatchSource:0}: Error finding container fe68e76ea504f811cd5a93e6d719c6958a0451bd89ecec7ed32982fad4a98a71: Status 404 returned error can't find the container with id fe68e76ea504f811cd5a93e6d719c6958a0451bd89ecec7ed32982fad4a98a71
Jul 15 10:45:34.138004 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:34.137980    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af7557c_6bfb_4ce6_a879_f95ca89cabdf.slice/crio-fdf4247ed9b844c88eeb135d03fc0a373c544d78ec846239c12b41bd0bff12e8 WatchSource:0}: Error finding container fdf4247ed9b844c88eeb135d03fc0a373c544d78ec846239c12b41bd0bff12e8: Status 404 returned error can't find the container with id fdf4247ed9b844c88eeb135d03fc0a373c544d78ec846239c12b41bd0bff12e8
Jul 15 10:45:34.139599 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:34.139580    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfb31e6_b8ba_4726_afb8_6892dd4adcc7.slice/crio-2e78d7d102574e23a0110589aa53a9833d15a58dc882a949bb739136d3ee7bd5 WatchSource:0}: Error finding container 2e78d7d102574e23a0110589aa53a9833d15a58dc882a949bb739136d3ee7bd5: Status 404 returned error can't find the container with id 2e78d7d102574e23a0110589aa53a9833d15a58dc882a949bb739136d3ee7bd5
Jul 15 10:45:34.141057 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:45:34.140911    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b844b5_2671_4bfa_b242_40537d33c32f.slice/crio-e69e2116b8c1649aaa52ba4514db66425f71f0df928bee8462bb50af1c6a6008 WatchSource:0}: Error finding container e69e2116b8c1649aaa52ba4514db66425f71f0df928bee8462bb50af1c6a6008: Status 404 returned error can't find the container with id e69e2116b8c1649aaa52ba4514db66425f71f0df928bee8462bb50af1c6a6008
Jul 15 10:45:34.196163 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:34.196116    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 10:40:33 +0000 UTC" deadline="2028-01-29 09:39:36.952696419 +0000 UTC"
Jul 15 10:45:34.196163 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:34.196160    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13510h54m2.756539351s"
Jul 15 10:45:34.272704 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:34.272675    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wcspb" event={"ID":"cdfb31e6-b8ba-4726-afb8-6892dd4adcc7","Type":"ContainerStarted","Data":"2e78d7d102574e23a0110589aa53a9833d15a58dc882a949bb739136d3ee7bd5"}
Jul 15 10:45:34.274267 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:34.274243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hxflc" event={"ID":"c118fdb2-4d52-4883-99c6-cf8609b6d8dd","Type":"ContainerStarted","Data":"9bf094358b764977c4950849b8051626f0dc0244c774f6c1c8e6f24b93236145"}
Jul 15 10:45:34.276024 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:34.275998    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ww56m" event={"ID":"6918f75f-21cc-47c8-9332-c1055bdbf5df","Type":"ContainerStarted","Data":"405b19862aa8c1b3661c900f6f2137f617a7221212e397d3c5f97ec0b9fcd858"}
Jul 15 10:45:34.277340 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:34.277309    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xtlgc" event={"ID":"6af7557c-6bfb-4ce6-a879-f95ca89cabdf","Type":"ContainerStarted","Data":"fdf4247ed9b844c88eeb135d03fc0a373c544d78ec846239c12b41bd0bff12e8"}
Jul 15 10:45:34.278376 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:34.278353    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7" event={"ID":"63ae8640-95c8-4f02-a7a9-df431281ef52","Type":"ContainerStarted","Data":"fe68e76ea504f811cd5a93e6d719c6958a0451bd89ecec7ed32982fad4a98a71"}
Jul 15 10:45:34.280251 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:34.280231    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal" event={"ID":"87afa8806d81c1e28dbf5cedd9e00a8e","Type":"ContainerStarted","Data":"bdaedcef8d33bd6de8590744dbe631913370e6ffa3318b72f9ecfd708db4639a"}
Jul 15 10:45:34.281346 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:34.281325    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw" event={"ID":"20239449-cde8-4cb1-9c0f-dad9abb7cd58","Type":"ContainerStarted","Data":"955d108902d2e4c870f0ab870f01f43d448a2cb0d8478270f4a170b59b4b4897"}
Jul 15 10:45:34.283358 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:34.283335    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-68n9x" event={"ID":"78b844b5-2671-4bfa-b242-40537d33c32f","Type":"ContainerStarted","Data":"e69e2116b8c1649aaa52ba4514db66425f71f0df928bee8462bb50af1c6a6008"}
Jul 15 10:45:34.284470 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:34.284448    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2jprt" event={"ID":"4b240a0c-d42c-4dba-b5ea-9be576447190","Type":"ContainerStarted","Data":"817e5be38617128713f6660e925fcba4cf794a7147631ae4b632d80aaf9b35b9"}
Jul 15 10:45:34.285553 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:34.285534    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4qs25" event={"ID":"6321df80-acdb-482c-bb4e-d04645f9131f","Type":"ContainerStarted","Data":"bbc6e493e81caf227fdd35d36242de369fff225f38fd97151998ae67d321a049"}
Jul 15 10:45:34.749970 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:34.749891    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret\") pod \"global-pull-secret-syncer-5ffl8\" (UID: \"dae7d2b5-9b3e-4ffe-85a3-087ad2313052\") " pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:34.750142 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:34.750085    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:34.750203 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:34.750150    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret podName:dae7d2b5-9b3e-4ffe-85a3-087ad2313052 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:36.750132694 +0000 UTC m=+5.980100209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret") pod "global-pull-secret-syncer-5ffl8" (UID: "dae7d2b5-9b3e-4ffe-85a3-087ad2313052") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:34.850609 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:34.850569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjtrg\" (UniqueName: \"kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg\") pod \"network-check-target-m7s5x\" (UID: \"11345472-e392-4a24-b60c-43b5fe6dae64\") " pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:34.850765 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:34.850628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs\") pod \"network-metrics-daemon-hkvw8\" (UID: \"8a1c4669-2697-411f-80ff-c3c5819361dc\") " pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:34.850835 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:34.850774    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:34.850892 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:34.850838    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs podName:8a1c4669-2697-411f-80ff-c3c5819361dc nodeName:}" failed. No retries permitted until 2026-07-15 10:45:36.850818313 +0000 UTC m=+6.080785830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs") pod "network-metrics-daemon-hkvw8" (UID: "8a1c4669-2697-411f-80ff-c3c5819361dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:34.851271 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:34.851250    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:45:34.851336 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:34.851285    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:45:34.851336 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:34.851299    2566 projected.go:196] Error preparing data for projected volume kube-api-access-tjtrg for pod openshift-network-diagnostics/network-check-target-m7s5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:34.851433 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:34.851347    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg podName:11345472-e392-4a24-b60c-43b5fe6dae64 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:36.851331069 +0000 UTC m=+6.081298581 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tjtrg" (UniqueName: "kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg") pod "network-check-target-m7s5x" (UID: "11345472-e392-4a24-b60c-43b5fe6dae64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:35.196385 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:35.196246    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 10:40:33 +0000 UTC" deadline="2028-04-07 17:45:04.46756653 +0000 UTC"
Jul 15 10:45:35.196385 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:35.196286    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15174h59m29.271285079s"
Jul 15 10:45:35.268599 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:35.266625    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:35.268599 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:35.266749    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:45:35.268599 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:35.266783    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:35.268599 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:35.266903    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkvw8" podUID="8a1c4669-2697-411f-80ff-c3c5819361dc"
Jul 15 10:45:35.268599 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:35.266633    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:35.268599 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:35.266998    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:45:35.303932 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:35.303894    2566 generic.go:356] "Generic (PLEG): container finished" podID="395ae80f9c797fbcfa996f0e1f6c0bf1" containerID="6e1e8d3007c66f5d6e0f98d256d2386eef5d0702623f2cdb6408bda239621d94" exitCode=0
Jul 15 10:45:35.304850 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:35.304823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal" event={"ID":"395ae80f9c797fbcfa996f0e1f6c0bf1","Type":"ContainerDied","Data":"6e1e8d3007c66f5d6e0f98d256d2386eef5d0702623f2cdb6408bda239621d94"}
Jul 15 10:45:35.363885 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:35.363832    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-26.ec2.internal" podStartSLOduration=3.363812304 podStartE2EDuration="3.363812304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:34.305714999 +0000 UTC m=+3.535682530" watchObservedRunningTime="2026-07-15 10:45:35.363812304 +0000 UTC m=+4.593779836"
Jul 15 10:45:36.319959 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:36.319918    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal" event={"ID":"395ae80f9c797fbcfa996f0e1f6c0bf1","Type":"ContainerStarted","Data":"11f22216a0746a674929829e4e5ca5e1b387f36bdea78e24eb7d0480af529a2a"}
Jul 15 10:45:36.766350 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:36.766268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret\") pod \"global-pull-secret-syncer-5ffl8\" (UID: \"dae7d2b5-9b3e-4ffe-85a3-087ad2313052\") " pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:36.766524 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:36.766454    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:36.766591 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:36.766536    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret podName:dae7d2b5-9b3e-4ffe-85a3-087ad2313052 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:40.76651763 +0000 UTC m=+9.996485143 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret") pod "global-pull-secret-syncer-5ffl8" (UID: "dae7d2b5-9b3e-4ffe-85a3-087ad2313052") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:36.867136 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:36.867095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjtrg\" (UniqueName: \"kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg\") pod \"network-check-target-m7s5x\" (UID: \"11345472-e392-4a24-b60c-43b5fe6dae64\") " pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:36.867312 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:36.867152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs\") pod \"network-metrics-daemon-hkvw8\" (UID: \"8a1c4669-2697-411f-80ff-c3c5819361dc\") " pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:36.867312 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:36.867295    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:36.867419 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:36.867355    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs podName:8a1c4669-2697-411f-80ff-c3c5819361dc nodeName:}" failed. No retries permitted until 2026-07-15 10:45:40.867337157 +0000 UTC m=+10.097304680 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs") pod "network-metrics-daemon-hkvw8" (UID: "8a1c4669-2697-411f-80ff-c3c5819361dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:36.867794 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:36.867774    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:45:36.867892 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:36.867798    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:45:36.867892 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:36.867812    2566 projected.go:196] Error preparing data for projected volume kube-api-access-tjtrg for pod openshift-network-diagnostics/network-check-target-m7s5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:36.867892 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:36.867861    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg podName:11345472-e392-4a24-b60c-43b5fe6dae64 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:40.867844916 +0000 UTC m=+10.097812428 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tjtrg" (UniqueName: "kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg") pod "network-check-target-m7s5x" (UID: "11345472-e392-4a24-b60c-43b5fe6dae64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:37.262711 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:37.262627    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:37.262869 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:37.262765    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkvw8" podUID="8a1c4669-2697-411f-80ff-c3c5819361dc"
Jul 15 10:45:37.262869 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:37.262628    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:37.262869 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:37.262820    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:37.263019 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:37.262929    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:45:37.263019 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:37.263005    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:45:39.264677 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:39.264604    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:39.264677 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:39.264648    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:39.264677 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:39.264669    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:39.265206 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:39.264820    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:45:39.265302 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:39.265274    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:45:39.265390 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:39.265379    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkvw8" podUID="8a1c4669-2697-411f-80ff-c3c5819361dc"
Jul 15 10:45:40.801115 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:40.801072    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret\") pod \"global-pull-secret-syncer-5ffl8\" (UID: \"dae7d2b5-9b3e-4ffe-85a3-087ad2313052\") " pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:40.801544 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:40.801214    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:40.801544 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:40.801275    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret podName:dae7d2b5-9b3e-4ffe-85a3-087ad2313052 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:48.801258114 +0000 UTC m=+18.031225624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret") pod "global-pull-secret-syncer-5ffl8" (UID: "dae7d2b5-9b3e-4ffe-85a3-087ad2313052") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:40.901800 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:40.901763    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjtrg\" (UniqueName: \"kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg\") pod \"network-check-target-m7s5x\" (UID: \"11345472-e392-4a24-b60c-43b5fe6dae64\") " pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:40.901977 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:40.901811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs\") pod \"network-metrics-daemon-hkvw8\" (UID: \"8a1c4669-2697-411f-80ff-c3c5819361dc\") " pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:40.901977 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:40.901929    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:40.901977 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:40.901959    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:45:40.901977 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:40.901976    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs podName:8a1c4669-2697-411f-80ff-c3c5819361dc nodeName:}" failed. No retries permitted until 2026-07-15 10:45:48.901962744 +0000 UTC m=+18.131930253 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs") pod "network-metrics-daemon-hkvw8" (UID: "8a1c4669-2697-411f-80ff-c3c5819361dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:40.902208 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:40.901983    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:45:40.902208 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:40.901997    2566 projected.go:196] Error preparing data for projected volume kube-api-access-tjtrg for pod openshift-network-diagnostics/network-check-target-m7s5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:40.902208 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:40.902048    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg podName:11345472-e392-4a24-b60c-43b5fe6dae64 nodeName:}" failed. No retries permitted until 2026-07-15 10:45:48.902031663 +0000 UTC m=+18.131999185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tjtrg" (UniqueName: "kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg") pod "network-check-target-m7s5x" (UID: "11345472-e392-4a24-b60c-43b5fe6dae64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:41.262701 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:41.262672    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:41.262855 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:41.262782    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:45:41.263176 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:41.263159    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:41.263289 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:41.263268    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkvw8" podUID="8a1c4669-2697-411f-80ff-c3c5819361dc"
Jul 15 10:45:41.263344 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:41.263333    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:41.263414 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:41.263399    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:45:43.263480 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:43.263451    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:43.263954 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:43.263601    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:43.263954 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:43.263600    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:45:43.263954 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:43.263713    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:45:43.263954 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:43.263761    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:43.263954 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:43.263845    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkvw8" podUID="8a1c4669-2697-411f-80ff-c3c5819361dc"
Jul 15 10:45:45.262455 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:45.262225    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:45.262902 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:45.262221    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:45.262902 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:45.262566    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:45:45.262902 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:45.262285    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:45.262902 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:45.262639    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkvw8" podUID="8a1c4669-2697-411f-80ff-c3c5819361dc"
Jul 15 10:45:45.262902 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:45.262722    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:45:47.262160 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:47.262122    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:47.262611 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:47.262128    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:47.262693 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:47.262656    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkvw8" podUID="8a1c4669-2697-411f-80ff-c3c5819361dc"
Jul 15 10:45:47.262822 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:47.262789    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:45:47.262921 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:47.262133    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:47.263072 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:47.263044    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:45:48.861154 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:48.861113    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret\") pod \"global-pull-secret-syncer-5ffl8\" (UID: \"dae7d2b5-9b3e-4ffe-85a3-087ad2313052\") " pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:48.861624 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:48.861248    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:48.861624 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:48.861317    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret podName:dae7d2b5-9b3e-4ffe-85a3-087ad2313052 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:04.861299945 +0000 UTC m=+34.091267473 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret") pod "global-pull-secret-syncer-5ffl8" (UID: "dae7d2b5-9b3e-4ffe-85a3-087ad2313052") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:45:48.961753 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:48.961715    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjtrg\" (UniqueName: \"kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg\") pod \"network-check-target-m7s5x\" (UID: \"11345472-e392-4a24-b60c-43b5fe6dae64\") " pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:48.961753 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:48.961760    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs\") pod \"network-metrics-daemon-hkvw8\" (UID: \"8a1c4669-2697-411f-80ff-c3c5819361dc\") " pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:48.962033 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:48.961877    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:48.962033 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:48.961902    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:45:48.962033 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:48.961922    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:45:48.962033 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:48.961933    2566 projected.go:196] Error preparing data for projected volume kube-api-access-tjtrg for pod openshift-network-diagnostics/network-check-target-m7s5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:48.962033 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:48.961945    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs podName:8a1c4669-2697-411f-80ff-c3c5819361dc nodeName:}" failed. No retries permitted until 2026-07-15 10:46:04.961924588 +0000 UTC m=+34.191892111 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs") pod "network-metrics-daemon-hkvw8" (UID: "8a1c4669-2697-411f-80ff-c3c5819361dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:45:48.962033 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:48.961965    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg podName:11345472-e392-4a24-b60c-43b5fe6dae64 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:04.961955614 +0000 UTC m=+34.191923124 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tjtrg" (UniqueName: "kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg") pod "network-check-target-m7s5x" (UID: "11345472-e392-4a24-b60c-43b5fe6dae64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:45:49.261612 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:49.261578    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:49.261773 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:49.261703    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:45:49.261773 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:49.261582    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:49.261773 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:49.261730    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:49.261947 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:49.261793    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:45:49.261947 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:49.261872    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkvw8" podUID="8a1c4669-2697-411f-80ff-c3c5819361dc"
Jul 15 10:45:51.264195 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:51.264154    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:51.264637 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:51.264290    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkvw8" podUID="8a1c4669-2697-411f-80ff-c3c5819361dc"
Jul 15 10:45:51.264772 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:51.264710    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:51.264821 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:51.264805    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:45:51.264878 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:51.264863    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:51.264956 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:51.264939    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:45:53.262804 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.262573    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:53.263430 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.262580    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:53.263430 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:53.262906    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:45:53.263430 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.262606    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:53.263430 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:53.263002    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:45:53.263430 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:53.263073    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkvw8" podUID="8a1c4669-2697-411f-80ff-c3c5819361dc"
Jul 15 10:45:53.356810 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.356764    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 10:45:53.357096 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.357073    2566 generic.go:356] "Generic (PLEG): container finished" podID="20239449-cde8-4cb1-9c0f-dad9abb7cd58" containerID="0b9453d48686bd45aacd4778f268dfd05ab9d03d4375a27f41ade6f6ac8b8508" exitCode=1
Jul 15 10:45:53.357181 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.357135    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw" event={"ID":"20239449-cde8-4cb1-9c0f-dad9abb7cd58","Type":"ContainerStarted","Data":"73fe9a3ae4f4800692761fa2d789ebffa85d24b6d8f5531b7adc42a5302f90c7"}
Jul 15 10:45:53.357220 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.357194    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw" event={"ID":"20239449-cde8-4cb1-9c0f-dad9abb7cd58","Type":"ContainerStarted","Data":"8afee79e8cf6e9cc90e28b29ece3f0d920923113c39b4f46a63828e558affb42"}
Jul 15 10:45:53.357252 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.357235    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw" event={"ID":"20239449-cde8-4cb1-9c0f-dad9abb7cd58","Type":"ContainerStarted","Data":"63ea48b0a3ec0351453052a93ccd760fffece3bc1602b1b9aaa0e97a2389c1a6"}
Jul 15 10:45:53.357284 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.357275    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw" event={"ID":"20239449-cde8-4cb1-9c0f-dad9abb7cd58","Type":"ContainerDied","Data":"0b9453d48686bd45aacd4778f268dfd05ab9d03d4375a27f41ade6f6ac8b8508"}
Jul 15 10:45:53.357329 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.357315    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw" event={"ID":"20239449-cde8-4cb1-9c0f-dad9abb7cd58","Type":"ContainerStarted","Data":"b371519f7b1f8eb87119934ed86edc161cf39f45890dbb532ef741a9188544f1"}
Jul 15 10:45:53.358463 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.358444    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-68n9x" event={"ID":"78b844b5-2671-4bfa-b242-40537d33c32f","Type":"ContainerStarted","Data":"5a09cc57d1a68e447b6900ec355850b44bb426bc2281ac49b775069e648ab391"}
Jul 15 10:45:53.359799 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.359781    2566 generic.go:356] "Generic (PLEG): container finished" podID="6321df80-acdb-482c-bb4e-d04645f9131f" containerID="bb1693ebb2ca31e4382d320cc21eaab79c558cc4f0d358893ed699ee20e1c30e" exitCode=0
Jul 15 10:45:53.359891 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.359841    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4qs25" event={"ID":"6321df80-acdb-482c-bb4e-d04645f9131f","Type":"ContainerDied","Data":"bb1693ebb2ca31e4382d320cc21eaab79c558cc4f0d358893ed699ee20e1c30e"}
Jul 15 10:45:53.361254 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.361227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wcspb" event={"ID":"cdfb31e6-b8ba-4726-afb8-6892dd4adcc7","Type":"ContainerStarted","Data":"42686169caa6708506c48b9f7e1c645cef17815354e355f14da9e1665284079e"}
Jul 15 10:45:53.362546 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.362518    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hxflc" event={"ID":"c118fdb2-4d52-4883-99c6-cf8609b6d8dd","Type":"ContainerStarted","Data":"6c43fca9a9f295f9488f25809ad4a7b763b618972447c4d5454c3a493817787c"}
Jul 15 10:45:53.363862 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.363837    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ww56m" event={"ID":"6918f75f-21cc-47c8-9332-c1055bdbf5df","Type":"ContainerStarted","Data":"6f40d22ff53a57493591963c4636f0bc2c07eda1a51821095fc0643042340ae3"}
Jul 15 10:45:53.365213 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.365192    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xtlgc" event={"ID":"6af7557c-6bfb-4ce6-a879-f95ca89cabdf","Type":"ContainerStarted","Data":"113bdde8c1ebb53fa582c400fbd7acd3993a128ff39dc92d4f3e67a26deb70b5"}
Jul 15 10:45:53.366783 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.366602    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7" event={"ID":"63ae8640-95c8-4f02-a7a9-df431281ef52","Type":"ContainerStarted","Data":"a42a8f6bf2f0f902dc4cdd43bf4067b78ec47ab3b6751fa6265af59080a05fc4"}
Jul 15 10:45:53.386029 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.385991    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-26.ec2.internal" podStartSLOduration=21.385979784 podStartE2EDuration="21.385979784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:36.339186488 +0000 UTC m=+5.569154020" watchObservedRunningTime="2026-07-15 10:45:53.385979784 +0000 UTC m=+22.615947315"
Jul 15 10:45:53.400691 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.400655    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ww56m" podStartSLOduration=3.8443033140000002 podStartE2EDuration="22.400641252s" totalImagesPullingTime="18.556337938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:53.400391724 +0000 UTC m=+22.630359255" watchObservedRunningTime="2026-07-15 10:45:53.400641252 +0000 UTC m=+22.630608783"
Jul 15 10:45:53.415866 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.415828    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wcspb" podStartSLOduration=11.950535941 podStartE2EDuration="22.41581267s" totalImagesPullingTime="10.465276729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:53.415388748 +0000 UTC m=+22.645356278" watchObservedRunningTime="2026-07-15 10:45:53.41581267 +0000 UTC m=+22.645780203"
Jul 15 10:45:53.432181 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.432145    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-68n9x" podStartSLOduration=3.874365357 podStartE2EDuration="22.432128573s" totalImagesPullingTime="18.557763216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:53.432087557 +0000 UTC m=+22.662055089" watchObservedRunningTime="2026-07-15 10:45:53.432128573 +0000 UTC m=+22.662096107"
Jul 15 10:45:53.449380 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.449337    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xtlgc" podStartSLOduration=3.860928673 podStartE2EDuration="22.449320978s" totalImagesPullingTime="18.588392305s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:53.449280557 +0000 UTC m=+22.679248088" watchObservedRunningTime="2026-07-15 10:45:53.449320978 +0000 UTC m=+22.679288509"
Jul 15 10:45:53.463835 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:53.463790    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hxflc" podStartSLOduration=3.90802337 podStartE2EDuration="22.463772817s" totalImagesPullingTime="18.555749447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:53.463269485 +0000 UTC m=+22.693237017" watchObservedRunningTime="2026-07-15 10:45:53.463772817 +0000 UTC m=+22.693740348"
Jul 15 10:45:54.185415 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:54.185197    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 15 10:45:54.190701 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:54.190594    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T10:45:54.185420131Z","UUID":"8a937bff-a678-4d5e-9152-276478129cce","Handler":null,"Name":"","Endpoint":""}
Jul 15 10:45:54.191990 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:54.191970    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 15 10:45:54.192081 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:54.192013    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 15 10:45:54.371632 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:54.371549    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7" event={"ID":"63ae8640-95c8-4f02-a7a9-df431281ef52","Type":"ContainerStarted","Data":"4272ea0804950cf3a5c2e7a753b1989504dc26126030aa65d0e84894807b7a08"}
Jul 15 10:45:54.374608 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:54.374579    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 10:45:54.374981 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:54.374952    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw" event={"ID":"20239449-cde8-4cb1-9c0f-dad9abb7cd58","Type":"ContainerStarted","Data":"3d81cfcbdfe459a79cd5f71a26c5f1d99047676fb23e066d60970e140ebc8987"}
Jul 15 10:45:54.376734 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:54.376662    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2jprt" event={"ID":"4b240a0c-d42c-4dba-b5ea-9be576447190","Type":"ContainerStarted","Data":"10c57ef95ca98a74fdd56510172048307c627f8a3538ae07a762c231ab108405"}
Jul 15 10:45:55.262616 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:55.262530    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:55.262616 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:55.262576    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:55.262854 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:55.262541    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:55.262854 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:55.262683    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkvw8" podUID="8a1c4669-2697-411f-80ff-c3c5819361dc"
Jul 15 10:45:55.262854 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:55.262747    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:45:55.262854 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:55.262827    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:45:56.384735 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:56.384706    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 10:45:56.385369 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:56.385115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw" event={"ID":"20239449-cde8-4cb1-9c0f-dad9abb7cd58","Type":"ContainerStarted","Data":"834d6da217845ece7098bf99dff8c018a0f5b7ea0cbd5e7cc739d5cad5ce1f51"}
Jul 15 10:45:56.522349 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:56.522312    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wcspb"
Jul 15 10:45:56.523056 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:56.523035    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wcspb"
Jul 15 10:45:56.538362 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:56.538304    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2jprt" podStartSLOduration=6.968943352 podStartE2EDuration="25.538289172s" totalImagesPullingTime="18.56934582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:54.394740321 +0000 UTC m=+23.624707853" watchObservedRunningTime="2026-07-15 10:45:56.538289172 +0000 UTC m=+25.768256704"
Jul 15 10:45:57.262439 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:57.262403    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:57.262636 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:57.262555    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:57.262636 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:57.262566    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkvw8" podUID="8a1c4669-2697-411f-80ff-c3c5819361dc"
Jul 15 10:45:57.262729 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:57.262675    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:45:57.262729 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:57.262719    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:57.262804 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:57.262787    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:45:57.387761 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:57.387728    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wcspb"
Jul 15 10:45:57.388305 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:57.388285    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wcspb"
Jul 15 10:45:58.391670 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:58.391439    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7" event={"ID":"63ae8640-95c8-4f02-a7a9-df431281ef52","Type":"ContainerStarted","Data":"f9174061cf89c4f428329e60fa06de45dc97eedb438add15616845bbefceecc9"}
Jul 15 10:45:58.394731 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:58.394711    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 10:45:58.395101 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:58.395066    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw" event={"ID":"20239449-cde8-4cb1-9c0f-dad9abb7cd58","Type":"ContainerStarted","Data":"14c8096438d7ca8c0fcaa10163b4d6a15fbb589b9e281653713e0f185f98e16c"}
Jul 15 10:45:58.395408 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:58.395387    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:58.395520 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:58.395453    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:58.395520 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:58.395459    2566 scope.go:117] "RemoveContainer" containerID="0b9453d48686bd45aacd4778f268dfd05ab9d03d4375a27f41ade6f6ac8b8508"
Jul 15 10:45:58.395520 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:58.395514    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:58.397672 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:58.397641    2566 generic.go:356] "Generic (PLEG): container finished" podID="6321df80-acdb-482c-bb4e-d04645f9131f" containerID="d5840132dad3b04b70172e0fb4955d1211f74003e0bd2fd0cf747d24264ca10b" exitCode=0
Jul 15 10:45:58.397755 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:58.397700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4qs25" event={"ID":"6321df80-acdb-482c-bb4e-d04645f9131f","Type":"ContainerDied","Data":"d5840132dad3b04b70172e0fb4955d1211f74003e0bd2fd0cf747d24264ca10b"}
Jul 15 10:45:58.414603 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:58.414579    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:58.415033 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:58.415015    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:45:58.451798 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:58.451758    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5pkg7" podStartSLOduration=4.5104258 podStartE2EDuration="27.451743299s" totalImagesPullingTime="22.941317499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:58.415819711 +0000 UTC m=+27.645787243" watchObservedRunningTime="2026-07-15 10:45:58.451743299 +0000 UTC m=+27.681710887"
Jul 15 10:45:59.261906 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:59.261875    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:59.262062 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:59.261874    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:59.262062 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:59.261983    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkvw8" podUID="8a1c4669-2697-411f-80ff-c3c5819361dc"
Jul 15 10:45:59.262062 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:59.261874    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:59.262062 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:59.262047    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:45:59.262272 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:59.262110    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:45:59.403756 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:59.403732    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 10:45:59.404173 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:59.404152    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw" event={"ID":"20239449-cde8-4cb1-9c0f-dad9abb7cd58","Type":"ContainerStarted","Data":"5425da377403e08f7c4d28d6ff1d93b636c2a471217b4b11d9401019e002c47e"}
Jul 15 10:45:59.439367 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:59.439316    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw" podStartSLOduration=9.823462736 podStartE2EDuration="28.439299238s" totalImagesPullingTime="18.615836502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:45:59.438378771 +0000 UTC m=+28.668346301" watchObservedRunningTime="2026-07-15 10:45:59.439299238 +0000 UTC m=+28.669266770"
Jul 15 10:45:59.955647 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:59.955005    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m7s5x"]
Jul 15 10:45:59.955822 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:59.955738    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5ffl8"]
Jul 15 10:45:59.955822 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:59.955818    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:45:59.955888 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:59.955841    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:45:59.955920 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:59.955901    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:45:59.955991 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:59.955960    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:45:59.957689 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:59.957668    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hkvw8"]
Jul 15 10:45:59.957795 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:45:59.957776    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:45:59.957898 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:45:59.957880    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkvw8" podUID="8a1c4669-2697-411f-80ff-c3c5819361dc"
Jul 15 10:46:00.407865 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:00.407771    2566 generic.go:356] "Generic (PLEG): container finished" podID="6321df80-acdb-482c-bb4e-d04645f9131f" containerID="e40b5c831f9bb3b3b6713673ebe43601e0b6e79c933cb3696f8d41348db03353" exitCode=0
Jul 15 10:46:00.407865 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:00.407851    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4qs25" event={"ID":"6321df80-acdb-482c-bb4e-d04645f9131f","Type":"ContainerDied","Data":"e40b5c831f9bb3b3b6713673ebe43601e0b6e79c933cb3696f8d41348db03353"}
Jul 15 10:46:01.263086 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:01.263055    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:46:01.263240 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:01.263150    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:46:01.266086 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:01.263544    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:46:01.266086 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:01.263808    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:46:01.412308 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:01.412265    2566 generic.go:356] "Generic (PLEG): container finished" podID="6321df80-acdb-482c-bb4e-d04645f9131f" containerID="5175b30b6bd63cab9c1463c8cf15058dc92997fa9c5d2dad7e43b230a26e96ad" exitCode=0
Jul 15 10:46:01.412987 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:01.412314    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4qs25" event={"ID":"6321df80-acdb-482c-bb4e-d04645f9131f","Type":"ContainerDied","Data":"5175b30b6bd63cab9c1463c8cf15058dc92997fa9c5d2dad7e43b230a26e96ad"}
Jul 15 10:46:02.262243 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:02.262204    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:46:02.262385 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:02.262359    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkvw8" podUID="8a1c4669-2697-411f-80ff-c3c5819361dc"
Jul 15 10:46:03.262099 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:03.262063    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:46:03.262565 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:03.262065    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:46:03.262565 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:03.262188    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:46:03.262565 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:03.262272    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:46:04.262123 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:04.261880    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:46:04.262527 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:04.262223    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hkvw8" podUID="8a1c4669-2697-411f-80ff-c3c5819361dc"
Jul 15 10:46:04.887436 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:04.887390    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret\") pod \"global-pull-secret-syncer-5ffl8\" (UID: \"dae7d2b5-9b3e-4ffe-85a3-087ad2313052\") " pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:46:04.887663 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:04.887547    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:04.887663 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:04.887621    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret podName:dae7d2b5-9b3e-4ffe-85a3-087ad2313052 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:36.887602978 +0000 UTC m=+66.117570488 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret") pod "global-pull-secret-syncer-5ffl8" (UID: "dae7d2b5-9b3e-4ffe-85a3-087ad2313052") : object "kube-system"/"original-pull-secret" not registered
Jul 15 10:46:04.987755 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:04.987725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjtrg\" (UniqueName: \"kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg\") pod \"network-check-target-m7s5x\" (UID: \"11345472-e392-4a24-b60c-43b5fe6dae64\") " pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:46:04.987755 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:04.987761    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs\") pod \"network-metrics-daemon-hkvw8\" (UID: \"8a1c4669-2697-411f-80ff-c3c5819361dc\") " pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:46:04.987987 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:04.987892    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:04.987987 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:04.987906    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 15 10:46:04.987987 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:04.987925    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 15 10:46:04.987987 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:04.987936    2566 projected.go:196] Error preparing data for projected volume kube-api-access-tjtrg for pod openshift-network-diagnostics/network-check-target-m7s5x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:04.987987 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:04.987945    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs podName:8a1c4669-2697-411f-80ff-c3c5819361dc nodeName:}" failed. No retries permitted until 2026-07-15 10:46:36.987931279 +0000 UTC m=+66.217898787 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs") pod "network-metrics-daemon-hkvw8" (UID: "8a1c4669-2697-411f-80ff-c3c5819361dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 15 10:46:04.987987 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:04.987983    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg podName:11345472-e392-4a24-b60c-43b5fe6dae64 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:36.987970588 +0000 UTC m=+66.217938101 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tjtrg" (UniqueName: "kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg") pod "network-check-target-m7s5x" (UID: "11345472-e392-4a24-b60c-43b5fe6dae64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 15 10:46:05.271298 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.271269    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:46:05.271940 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.271327    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:46:05.272068 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:05.272027    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ffl8" podUID="dae7d2b5-9b3e-4ffe-85a3-087ad2313052"
Jul 15 10:46:05.272137 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:05.272058    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7s5x" podUID="11345472-e392-4a24-b60c-43b5fe6dae64"
Jul 15 10:46:05.580888 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.580823    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-26.ec2.internal" event="NodeReady"
Jul 15 10:46:05.581062 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.580969    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 15 10:46:05.627780 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.627754    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8f8hw"]
Jul 15 10:46:05.645624 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.645591    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b5jv4"]
Jul 15 10:46:05.645776 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.645732    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:05.648155 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.648129    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8cmqw\""
Jul 15 10:46:05.648299 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.648191    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 15 10:46:05.648365 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.648313    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 15 10:46:05.661941 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.661917    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8f8hw"]
Jul 15 10:46:05.662050 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.661950    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b5jv4"]
Jul 15 10:46:05.662050 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.662015    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b5jv4"
Jul 15 10:46:05.664685 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.664667    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 15 10:46:05.664803 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.664772    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 15 10:46:05.664803 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.664780    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 15 10:46:05.664803 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.664782    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-s2xwp\""
Jul 15 10:46:05.794081 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.794051    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg85x\" (UniqueName: \"kubernetes.io/projected/599648bd-bc11-4e5a-9664-0c4892ce9edc-kube-api-access-kg85x\") pod \"ingress-canary-b5jv4\" (UID: \"599648bd-bc11-4e5a-9664-0c4892ce9edc\") " pod="openshift-ingress-canary/ingress-canary-b5jv4"
Jul 15 10:46:05.794196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.794115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8556x\" (UniqueName: \"kubernetes.io/projected/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-kube-api-access-8556x\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:05.794196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.794154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/599648bd-bc11-4e5a-9664-0c4892ce9edc-cert\") pod \"ingress-canary-b5jv4\" (UID: \"599648bd-bc11-4e5a-9664-0c4892ce9edc\") " pod="openshift-ingress-canary/ingress-canary-b5jv4"
Jul 15 10:46:05.794196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.794177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:05.794292 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.794202    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-tmp-dir\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:05.794292 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.794242    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-config-volume\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:05.894878 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.894848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8556x\" (UniqueName: \"kubernetes.io/projected/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-kube-api-access-8556x\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:05.895008 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.894893    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/599648bd-bc11-4e5a-9664-0c4892ce9edc-cert\") pod \"ingress-canary-b5jv4\" (UID: \"599648bd-bc11-4e5a-9664-0c4892ce9edc\") " pod="openshift-ingress-canary/ingress-canary-b5jv4"
Jul 15 10:46:05.895008 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.894908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:05.895008 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.894933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-tmp-dir\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:05.895008 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.894972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-config-volume\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:05.895196 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:05.895023    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 10:46:05.895196 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:05.895089    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls podName:dfc3a194-a6c3-4cee-8f9c-225bc846d9b9 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:06.395068231 +0000 UTC m=+35.625035746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls") pod "dns-default-8f8hw" (UID: "dfc3a194-a6c3-4cee-8f9c-225bc846d9b9") : secret "dns-default-metrics-tls" not found
Jul 15 10:46:05.895196 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.895024    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg85x\" (UniqueName: \"kubernetes.io/projected/599648bd-bc11-4e5a-9664-0c4892ce9edc-kube-api-access-kg85x\") pod \"ingress-canary-b5jv4\" (UID: \"599648bd-bc11-4e5a-9664-0c4892ce9edc\") " pod="openshift-ingress-canary/ingress-canary-b5jv4"
Jul 15 10:46:05.895334 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.895312    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-tmp-dir\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:05.895756 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.895733    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-config-volume\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:05.899527 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.899485    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/599648bd-bc11-4e5a-9664-0c4892ce9edc-cert\") pod \"ingress-canary-b5jv4\" (UID: \"599648bd-bc11-4e5a-9664-0c4892ce9edc\") " pod="openshift-ingress-canary/ingress-canary-b5jv4"
Jul 15 10:46:05.905132 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.905099    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8556x\" (UniqueName: \"kubernetes.io/projected/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-kube-api-access-8556x\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:05.905442 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.905425    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg85x\" (UniqueName: \"kubernetes.io/projected/599648bd-bc11-4e5a-9664-0c4892ce9edc-kube-api-access-kg85x\") pod \"ingress-canary-b5jv4\" (UID: \"599648bd-bc11-4e5a-9664-0c4892ce9edc\") " pod="openshift-ingress-canary/ingress-canary-b5jv4"
Jul 15 10:46:05.970470 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:05.970446    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b5jv4"
Jul 15 10:46:06.148950 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:06.148871    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b5jv4"]
Jul 15 10:46:06.153376 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:46:06.153351    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599648bd_bc11_4e5a_9664_0c4892ce9edc.slice/crio-841db92cc82de9177d366985804e391c0b7ee344a525607272df84063611b470 WatchSource:0}: Error finding container 841db92cc82de9177d366985804e391c0b7ee344a525607272df84063611b470: Status 404 returned error can't find the container with id 841db92cc82de9177d366985804e391c0b7ee344a525607272df84063611b470
Jul 15 10:46:06.262315 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:06.262154    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:46:06.264940 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:06.264918    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 15 10:46:06.265064 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:06.265040    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vfsm8\""
Jul 15 10:46:06.400185 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:06.400106    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:06.400589 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:06.400276    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 10:46:06.400589 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:06.400338    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls podName:dfc3a194-a6c3-4cee-8f9c-225bc846d9b9 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:07.400322152 +0000 UTC m=+36.630289665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls") pod "dns-default-8f8hw" (UID: "dfc3a194-a6c3-4cee-8f9c-225bc846d9b9") : secret "dns-default-metrics-tls" not found
Jul 15 10:46:06.427579 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:06.427548    2566 generic.go:356] "Generic (PLEG): container finished" podID="6321df80-acdb-482c-bb4e-d04645f9131f" containerID="cf3acdc78a8ba063a62a39c515a3530329bc22d018bc87bf5f730def1143d002" exitCode=0
Jul 15 10:46:06.427751 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:06.427625    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4qs25" event={"ID":"6321df80-acdb-482c-bb4e-d04645f9131f","Type":"ContainerDied","Data":"cf3acdc78a8ba063a62a39c515a3530329bc22d018bc87bf5f730def1143d002"}
Jul 15 10:46:06.428727 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:06.428651    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b5jv4" event={"ID":"599648bd-bc11-4e5a-9664-0c4892ce9edc","Type":"ContainerStarted","Data":"841db92cc82de9177d366985804e391c0b7ee344a525607272df84063611b470"}
Jul 15 10:46:07.262296 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:07.262263    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:46:07.262590 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:07.262519    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:46:07.265864 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:07.265685    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 15 10:46:07.272586 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:07.272562    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 15 10:46:07.273152 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:07.273130    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xhf7j\""
Jul 15 10:46:07.273693 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:07.273676    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 15 10:46:07.407954 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:07.407918    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:07.408488 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:07.408078    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 10:46:07.408488 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:07.408166    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls podName:dfc3a194-a6c3-4cee-8f9c-225bc846d9b9 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:09.408144327 +0000 UTC m=+38.638111850 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls") pod "dns-default-8f8hw" (UID: "dfc3a194-a6c3-4cee-8f9c-225bc846d9b9") : secret "dns-default-metrics-tls" not found
Jul 15 10:46:07.435423 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:07.435392    2566 generic.go:356] "Generic (PLEG): container finished" podID="6321df80-acdb-482c-bb4e-d04645f9131f" containerID="7fe0c43367f19356d533c635b0c6459f51762b7b3b3949b58e46d730ff223b84" exitCode=0
Jul 15 10:46:07.435603 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:07.435440    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4qs25" event={"ID":"6321df80-acdb-482c-bb4e-d04645f9131f","Type":"ContainerDied","Data":"7fe0c43367f19356d533c635b0c6459f51762b7b3b3949b58e46d730ff223b84"}
Jul 15 10:46:08.444218 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:08.444180    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4qs25" event={"ID":"6321df80-acdb-482c-bb4e-d04645f9131f","Type":"ContainerStarted","Data":"5c60fa5e11498db2031df280289aee5f409bc5b8d3130b6c4fa5b53612e3935c"}
Jul 15 10:46:09.426327 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:09.426296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:09.426485 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:09.426443    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 10:46:09.426556 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:09.426523    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls podName:dfc3a194-a6c3-4cee-8f9c-225bc846d9b9 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:13.426491455 +0000 UTC m=+42.656458964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls") pod "dns-default-8f8hw" (UID: "dfc3a194-a6c3-4cee-8f9c-225bc846d9b9") : secret "dns-default-metrics-tls" not found
Jul 15 10:46:09.448399 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:09.448364    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b5jv4" event={"ID":"599648bd-bc11-4e5a-9664-0c4892ce9edc","Type":"ContainerStarted","Data":"6d1ba124bed259b1c5118eac52da355a16ac8a4d33d53c3619a128eb1a4ded06"}
Jul 15 10:46:09.475130 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:09.475081    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b5jv4" podStartSLOduration=1.688863697 podStartE2EDuration="4.475062642s" totalImagesPullingTime="2.786198945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:09.475040982 +0000 UTC m=+38.705008513" watchObservedRunningTime="2026-07-15 10:46:09.475062642 +0000 UTC m=+38.705030174"
Jul 15 10:46:09.475382 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:09.475360    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4qs25" podStartSLOduration=13.584513047 podStartE2EDuration="38.475354062s" totalImagesPullingTime="24.890841015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:08.477046512 +0000 UTC m=+37.707014046" watchObservedRunningTime="2026-07-15 10:46:09.475354062 +0000 UTC m=+38.705321590"
Jul 15 10:46:13.454174 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:13.453995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:13.454174 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:13.454149    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 10:46:13.454711 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:13.454233    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls podName:dfc3a194-a6c3-4cee-8f9c-225bc846d9b9 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:21.454211982 +0000 UTC m=+50.684179490 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls") pod "dns-default-8f8hw" (UID: "dfc3a194-a6c3-4cee-8f9c-225bc846d9b9") : secret "dns-default-metrics-tls" not found
Jul 15 10:46:21.513138 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:21.513105    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:21.513519 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:21.513215    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 10:46:21.513519 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:21.513271    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls podName:dfc3a194-a6c3-4cee-8f9c-225bc846d9b9 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:37.51325847 +0000 UTC m=+66.743225980 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls") pod "dns-default-8f8hw" (UID: "dfc3a194-a6c3-4cee-8f9c-225bc846d9b9") : secret "dns-default-metrics-tls" not found
Jul 15 10:46:28.093292 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.093254    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt"]
Jul 15 10:46:28.095022 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.095005    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw"]
Jul 15 10:46:28.095155 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.095137    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt"
Jul 15 10:46:28.096506 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.096469    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw"
Jul 15 10:46:28.097816 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.097796    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 15 10:46:28.098881 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.098863    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 15 10:46:28.098967 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.098906    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 15 10:46:28.098967 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.098915    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 15 10:46:28.098967 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.098941    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-bw2z6\""
Jul 15 10:46:28.099174 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.099159    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 15 10:46:28.099470 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.099458    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 15 10:46:28.100681 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.100664    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mx2pm\""
Jul 15 10:46:28.101302 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.101288    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 15 10:46:28.115571 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.115421    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt"]
Jul 15 10:46:28.116351 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.116247    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw"]
Jul 15 10:46:28.162552 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.162525    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2df82869-758b-450d-bec7-14b4e125ec49-serving-cert\") pod \"service-ca-operator-7695fc6c8c-r7qfw\" (UID: \"2df82869-758b-450d-bec7-14b4e125ec49\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw"
Jul 15 10:46:28.162552 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.162552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2df82869-758b-450d-bec7-14b4e125ec49-config\") pod \"service-ca-operator-7695fc6c8c-r7qfw\" (UID: \"2df82869-758b-450d-bec7-14b4e125ec49\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw"
Jul 15 10:46:28.162752 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.162573    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z64jk\" (UniqueName: \"kubernetes.io/projected/2df82869-758b-450d-bec7-14b4e125ec49-kube-api-access-z64jk\") pod \"service-ca-operator-7695fc6c8c-r7qfw\" (UID: \"2df82869-758b-450d-bec7-14b4e125ec49\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw"
Jul 15 10:46:28.162752 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.162660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-m9qxt\" (UID: \"bc05cca4-8f13-4ee6-8673-a1dd0a651776\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt"
Jul 15 10:46:28.162752 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.162688    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkgjk\" (UniqueName: \"kubernetes.io/projected/bc05cca4-8f13-4ee6-8673-a1dd0a651776-kube-api-access-rkgjk\") pod \"cluster-samples-operator-65db858894-m9qxt\" (UID: \"bc05cca4-8f13-4ee6-8673-a1dd0a651776\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt"
Jul 15 10:46:28.263075 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.263049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-m9qxt\" (UID: \"bc05cca4-8f13-4ee6-8673-a1dd0a651776\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt"
Jul 15 10:46:28.263214 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.263081    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkgjk\" (UniqueName: \"kubernetes.io/projected/bc05cca4-8f13-4ee6-8673-a1dd0a651776-kube-api-access-rkgjk\") pod \"cluster-samples-operator-65db858894-m9qxt\" (UID: \"bc05cca4-8f13-4ee6-8673-a1dd0a651776\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt"
Jul 15 10:46:28.263214 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.263146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2df82869-758b-450d-bec7-14b4e125ec49-serving-cert\") pod \"service-ca-operator-7695fc6c8c-r7qfw\" (UID: \"2df82869-758b-450d-bec7-14b4e125ec49\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw"
Jul 15 10:46:28.263214 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.263169    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2df82869-758b-450d-bec7-14b4e125ec49-config\") pod \"service-ca-operator-7695fc6c8c-r7qfw\" (UID: \"2df82869-758b-450d-bec7-14b4e125ec49\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw"
Jul 15 10:46:28.263214 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.263194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z64jk\" (UniqueName: \"kubernetes.io/projected/2df82869-758b-450d-bec7-14b4e125ec49-kube-api-access-z64jk\") pod \"service-ca-operator-7695fc6c8c-r7qfw\" (UID: \"2df82869-758b-450d-bec7-14b4e125ec49\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw"
Jul 15 10:46:28.263449 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:28.263214    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 15 10:46:28.263449 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:28.263277    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls podName:bc05cca4-8f13-4ee6-8673-a1dd0a651776 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:28.763259735 +0000 UTC m=+57.993227245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls") pod "cluster-samples-operator-65db858894-m9qxt" (UID: "bc05cca4-8f13-4ee6-8673-a1dd0a651776") : secret "samples-operator-tls" not found
Jul 15 10:46:28.263790 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.263767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2df82869-758b-450d-bec7-14b4e125ec49-config\") pod \"service-ca-operator-7695fc6c8c-r7qfw\" (UID: \"2df82869-758b-450d-bec7-14b4e125ec49\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw"
Jul 15 10:46:28.266738 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.266712    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2df82869-758b-450d-bec7-14b4e125ec49-serving-cert\") pod \"service-ca-operator-7695fc6c8c-r7qfw\" (UID: \"2df82869-758b-450d-bec7-14b4e125ec49\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw"
Jul 15 10:46:28.279529 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.279487    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z64jk\" (UniqueName: \"kubernetes.io/projected/2df82869-758b-450d-bec7-14b4e125ec49-kube-api-access-z64jk\") pod \"service-ca-operator-7695fc6c8c-r7qfw\" (UID: \"2df82869-758b-450d-bec7-14b4e125ec49\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw"
Jul 15 10:46:28.282190 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.282166    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkgjk\" (UniqueName: \"kubernetes.io/projected/bc05cca4-8f13-4ee6-8673-a1dd0a651776-kube-api-access-rkgjk\") pod \"cluster-samples-operator-65db858894-m9qxt\" (UID: \"bc05cca4-8f13-4ee6-8673-a1dd0a651776\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt"
Jul 15 10:46:28.409965 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.409890    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw"
Jul 15 10:46:28.532660 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.532629    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw"]
Jul 15 10:46:28.535689 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:46:28.535660    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df82869_758b_450d_bec7_14b4e125ec49.slice/crio-c8262f32051cc4190de1904579f8051ce15b4f3dc04c82fd2d5424884525a1b2 WatchSource:0}: Error finding container c8262f32051cc4190de1904579f8051ce15b4f3dc04c82fd2d5424884525a1b2: Status 404 returned error can't find the container with id c8262f32051cc4190de1904579f8051ce15b4f3dc04c82fd2d5424884525a1b2
Jul 15 10:46:28.770148 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:28.770115    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-m9qxt\" (UID: \"bc05cca4-8f13-4ee6-8673-a1dd0a651776\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt"
Jul 15 10:46:28.770306 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:28.770286    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 15 10:46:28.770368 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:28.770357    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls podName:bc05cca4-8f13-4ee6-8673-a1dd0a651776 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:29.770341649 +0000 UTC m=+59.000309158 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls") pod "cluster-samples-operator-65db858894-m9qxt" (UID: "bc05cca4-8f13-4ee6-8673-a1dd0a651776") : secret "samples-operator-tls" not found
Jul 15 10:46:29.494588 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:29.494549    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw" event={"ID":"2df82869-758b-450d-bec7-14b4e125ec49","Type":"ContainerStarted","Data":"c8262f32051cc4190de1904579f8051ce15b4f3dc04c82fd2d5424884525a1b2"}
Jul 15 10:46:29.777886 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:29.777785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-m9qxt\" (UID: \"bc05cca4-8f13-4ee6-8673-a1dd0a651776\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt"
Jul 15 10:46:29.778054 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:29.777951    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 15 10:46:29.778054 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:29.778022    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls podName:bc05cca4-8f13-4ee6-8673-a1dd0a651776 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:31.778004317 +0000 UTC m=+61.007971825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls") pod "cluster-samples-operator-65db858894-m9qxt" (UID: "bc05cca4-8f13-4ee6-8673-a1dd0a651776") : secret "samples-operator-tls" not found
Jul 15 10:46:30.418628 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:30.418596    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5tqtw"
Jul 15 10:46:31.500719 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:31.500688    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw" event={"ID":"2df82869-758b-450d-bec7-14b4e125ec49","Type":"ContainerStarted","Data":"7f58bf9e46e2fd1b6db14cf79cb0d23326bdfe5d134aabb483f2302b3acc602f"}
Jul 15 10:46:31.523577 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:31.523530    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw" podStartSLOduration=1.014224923 podStartE2EDuration="3.52351745s" totalImagesPullingTime="2.509292527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:31.522257827 +0000 UTC m=+60.752225373" watchObservedRunningTime="2026-07-15 10:46:31.52351745 +0000 UTC m=+60.753484980"
Jul 15 10:46:31.793718 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:31.793641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-m9qxt\" (UID: \"bc05cca4-8f13-4ee6-8673-a1dd0a651776\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt"
Jul 15 10:46:31.793857 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:31.793814    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 15 10:46:31.793906 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:31.793888    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls podName:bc05cca4-8f13-4ee6-8673-a1dd0a651776 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:35.793871718 +0000 UTC m=+65.023839227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls") pod "cluster-samples-operator-65db858894-m9qxt" (UID: "bc05cca4-8f13-4ee6-8673-a1dd0a651776") : secret "samples-operator-tls" not found
Jul 15 10:46:32.368841 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:32.368803    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-drzdz"]
Jul 15 10:46:32.372136 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:32.372119    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-drzdz"
Jul 15 10:46:32.374945 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:32.374921    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 15 10:46:32.375043 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:32.374968    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 15 10:46:32.375749 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:32.375733    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-chscd\""
Jul 15 10:46:32.386334 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:32.386316    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-drzdz"]
Jul 15 10:46:32.398723 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:32.398701    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thvqk\" (UniqueName: \"kubernetes.io/projected/0b55cf39-c58b-4ef8-82b1-706516c52fc9-kube-api-access-thvqk\") pod \"migrator-7f59b54f6c-drzdz\" (UID: \"0b55cf39-c58b-4ef8-82b1-706516c52fc9\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-drzdz"
Jul 15 10:46:32.500052 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:32.500010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thvqk\" (UniqueName: \"kubernetes.io/projected/0b55cf39-c58b-4ef8-82b1-706516c52fc9-kube-api-access-thvqk\") pod \"migrator-7f59b54f6c-drzdz\" (UID: \"0b55cf39-c58b-4ef8-82b1-706516c52fc9\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-drzdz"
Jul 15 10:46:32.509141 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:32.509113    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thvqk\" (UniqueName: \"kubernetes.io/projected/0b55cf39-c58b-4ef8-82b1-706516c52fc9-kube-api-access-thvqk\") pod \"migrator-7f59b54f6c-drzdz\" (UID: \"0b55cf39-c58b-4ef8-82b1-706516c52fc9\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-drzdz"
Jul 15 10:46:32.681445 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:32.681375    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-drzdz"
Jul 15 10:46:32.803695 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:32.803620    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-drzdz"]
Jul 15 10:46:32.807512 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:46:32.807475    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b55cf39_c58b_4ef8_82b1_706516c52fc9.slice/crio-8baaced3555ed1993797dafd6f1a4aeb00459217bce5da291a5819d1c5da3490 WatchSource:0}: Error finding container 8baaced3555ed1993797dafd6f1a4aeb00459217bce5da291a5819d1c5da3490: Status 404 returned error can't find the container with id 8baaced3555ed1993797dafd6f1a4aeb00459217bce5da291a5819d1c5da3490
Jul 15 10:46:33.507394 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:33.507354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-drzdz" event={"ID":"0b55cf39-c58b-4ef8-82b1-706516c52fc9","Type":"ContainerStarted","Data":"8baaced3555ed1993797dafd6f1a4aeb00459217bce5da291a5819d1c5da3490"}
Jul 15 10:46:34.135059 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.134986    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t8tvc"]
Jul 15 10:46:34.137999 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.137983    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:34.152664 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.152644    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 15 10:46:34.152664 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.152654    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 15 10:46:34.152980 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.152966    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 15 10:46:34.153678 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.153663    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 15 10:46:34.153736 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.153719    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zgdp7\""
Jul 15 10:46:34.163084 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.163065    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t8tvc"]
Jul 15 10:46:34.210950 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.210919    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d30589a9-b904-4874-b34b-93d1ee5507d2-data-volume\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:34.211103 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.210959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:34.211103 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.211074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:34.211182 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.211117    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d30589a9-b904-4874-b34b-93d1ee5507d2-crio-socket\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:34.211223 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.211185    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw27p\" (UniqueName: \"kubernetes.io/projected/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-api-access-sw27p\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:34.315563 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.313183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:34.315563 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.313256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d30589a9-b904-4874-b34b-93d1ee5507d2-crio-socket\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:34.315563 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.313314    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw27p\" (UniqueName: \"kubernetes.io/projected/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-api-access-sw27p\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:34.315563 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.313380    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d30589a9-b904-4874-b34b-93d1ee5507d2-data-volume\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:34.315563 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.313428    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:34.315563 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:34.313585    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:34.315563 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:34.313655    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls podName:d30589a9-b904-4874-b34b-93d1ee5507d2 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:34.813635401 +0000 UTC m=+64.043602910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t8tvc" (UID: "d30589a9-b904-4874-b34b-93d1ee5507d2") : secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:34.315563 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:34.313998    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 15 10:46:34.315563 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:34.314064    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm podName:d30589a9-b904-4874-b34b-93d1ee5507d2 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:34.814048213 +0000 UTC m=+64.044015743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t8tvc" (UID: "d30589a9-b904-4874-b34b-93d1ee5507d2") : configmap "kube-rbac-proxy" not found
Jul 15 10:46:34.315563 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.314134    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d30589a9-b904-4874-b34b-93d1ee5507d2-crio-socket\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:34.315563 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.314710    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d30589a9-b904-4874-b34b-93d1ee5507d2-data-volume\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:34.325530 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.325489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw27p\" (UniqueName: \"kubernetes.io/projected/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-api-access-sw27p\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:34.511943 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.511907    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-drzdz" event={"ID":"0b55cf39-c58b-4ef8-82b1-706516c52fc9","Type":"ContainerStarted","Data":"1d9714d4101ac0700023bc426cb78c848e0b5e6f11c1db378f29704bbcd58a5a"}
Jul 15 10:46:34.512095 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.512007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-drzdz" event={"ID":"0b55cf39-c58b-4ef8-82b1-706516c52fc9","Type":"ContainerStarted","Data":"7485a4350105f949c7f61846c23c874f4cc58469ed909d3aaa9af55d673908b2"}
Jul 15 10:46:34.527778 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.527739    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-drzdz" podStartSLOduration=1.4607468 podStartE2EDuration="2.527726316s" totalImagesPullingTime="1.066979516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:34.527370276 +0000 UTC m=+63.757337807" watchObservedRunningTime="2026-07-15 10:46:34.527726316 +0000 UTC m=+63.757693900"
Jul 15 10:46:34.818126 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.818037    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:34.818126 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:34.818098    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:34.818339 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:34.818171    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 15 10:46:34.818339 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:34.818217    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:34.818339 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:34.818225    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm podName:d30589a9-b904-4874-b34b-93d1ee5507d2 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:35.818211878 +0000 UTC m=+65.048179386 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t8tvc" (UID: "d30589a9-b904-4874-b34b-93d1ee5507d2") : configmap "kube-rbac-proxy" not found
Jul 15 10:46:34.818339 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:34.818309    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls podName:d30589a9-b904-4874-b34b-93d1ee5507d2 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:35.818287601 +0000 UTC m=+65.048255113 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t8tvc" (UID: "d30589a9-b904-4874-b34b-93d1ee5507d2") : secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:35.028051 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.028018    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-n2v2j"]
Jul 15 10:46:35.032147 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.032132    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-n2v2j"
Jul 15 10:46:35.035708 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.035685    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 15 10:46:35.035708 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.035698    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 15 10:46:35.035879 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.035699    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5mgrk\""
Jul 15 10:46:35.035879 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.035741    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 15 10:46:35.036405 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.036388    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 15 10:46:35.049238 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.049215    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-n2v2j"]
Jul 15 10:46:35.119722 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.119659    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plzv9\" (UniqueName: \"kubernetes.io/projected/9840a307-d416-4490-854b-2f5d2aa0549a-kube-api-access-plzv9\") pod \"service-ca-79f68b9f4f-n2v2j\" (UID: \"9840a307-d416-4490-854b-2f5d2aa0549a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n2v2j"
Jul 15 10:46:35.119722 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.119715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9840a307-d416-4490-854b-2f5d2aa0549a-signing-cabundle\") pod \"service-ca-79f68b9f4f-n2v2j\" (UID: \"9840a307-d416-4490-854b-2f5d2aa0549a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n2v2j"
Jul 15 10:46:35.119875 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.119819    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9840a307-d416-4490-854b-2f5d2aa0549a-signing-key\") pod \"service-ca-79f68b9f4f-n2v2j\" (UID: \"9840a307-d416-4490-854b-2f5d2aa0549a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n2v2j"
Jul 15 10:46:35.220970 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.220936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9840a307-d416-4490-854b-2f5d2aa0549a-signing-key\") pod \"service-ca-79f68b9f4f-n2v2j\" (UID: \"9840a307-d416-4490-854b-2f5d2aa0549a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n2v2j"
Jul 15 10:46:35.221441 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.221008    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plzv9\" (UniqueName: \"kubernetes.io/projected/9840a307-d416-4490-854b-2f5d2aa0549a-kube-api-access-plzv9\") pod \"service-ca-79f68b9f4f-n2v2j\" (UID: \"9840a307-d416-4490-854b-2f5d2aa0549a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n2v2j"
Jul 15 10:46:35.221441 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.221054    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9840a307-d416-4490-854b-2f5d2aa0549a-signing-cabundle\") pod \"service-ca-79f68b9f4f-n2v2j\" (UID: \"9840a307-d416-4490-854b-2f5d2aa0549a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n2v2j"
Jul 15 10:46:35.221780 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.221758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9840a307-d416-4490-854b-2f5d2aa0549a-signing-cabundle\") pod \"service-ca-79f68b9f4f-n2v2j\" (UID: \"9840a307-d416-4490-854b-2f5d2aa0549a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n2v2j"
Jul 15 10:46:35.223453 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.223434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9840a307-d416-4490-854b-2f5d2aa0549a-signing-key\") pod \"service-ca-79f68b9f4f-n2v2j\" (UID: \"9840a307-d416-4490-854b-2f5d2aa0549a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n2v2j"
Jul 15 10:46:35.229679 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.229660    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plzv9\" (UniqueName: \"kubernetes.io/projected/9840a307-d416-4490-854b-2f5d2aa0549a-kube-api-access-plzv9\") pod \"service-ca-79f68b9f4f-n2v2j\" (UID: \"9840a307-d416-4490-854b-2f5d2aa0549a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-n2v2j"
Jul 15 10:46:35.341596 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.341566    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-n2v2j"
Jul 15 10:46:35.466597 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.466564    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-n2v2j"]
Jul 15 10:46:35.469286 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:46:35.469245    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9840a307_d416_4490_854b_2f5d2aa0549a.slice/crio-0f39bc6b9efd8d36bc8c65661c4bfde62cf6b04bc58ef46d5bf7c561b0188e26 WatchSource:0}: Error finding container 0f39bc6b9efd8d36bc8c65661c4bfde62cf6b04bc58ef46d5bf7c561b0188e26: Status 404 returned error can't find the container with id 0f39bc6b9efd8d36bc8c65661c4bfde62cf6b04bc58ef46d5bf7c561b0188e26
Jul 15 10:46:35.515113 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.515091    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-n2v2j" event={"ID":"9840a307-d416-4490-854b-2f5d2aa0549a","Type":"ContainerStarted","Data":"0f39bc6b9efd8d36bc8c65661c4bfde62cf6b04bc58ef46d5bf7c561b0188e26"}
Jul 15 10:46:35.826298 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.826263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:35.826463 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.826345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-m9qxt\" (UID: \"bc05cca4-8f13-4ee6-8673-a1dd0a651776\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt"
Jul 15 10:46:35.826463 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:35.826374    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:35.826463 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:35.826408    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:35.826463 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:35.826456    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 15 10:46:35.826619 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:35.826480    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls podName:d30589a9-b904-4874-b34b-93d1ee5507d2 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:37.82646489 +0000 UTC m=+67.056432400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t8tvc" (UID: "d30589a9-b904-4874-b34b-93d1ee5507d2") : secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:35.826619 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:35.826518    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm podName:d30589a9-b904-4874-b34b-93d1ee5507d2 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:37.82648725 +0000 UTC m=+67.056454759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t8tvc" (UID: "d30589a9-b904-4874-b34b-93d1ee5507d2") : configmap "kube-rbac-proxy" not found
Jul 15 10:46:35.826619 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:35.826516    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 15 10:46:35.826619 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:35.826577    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls podName:bc05cca4-8f13-4ee6-8673-a1dd0a651776 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:43.826562948 +0000 UTC m=+73.056530466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls") pod "cluster-samples-operator-65db858894-m9qxt" (UID: "bc05cca4-8f13-4ee6-8673-a1dd0a651776") : secret "samples-operator-tls" not found
Jul 15 10:46:36.519484 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:36.519449    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-n2v2j" event={"ID":"9840a307-d416-4490-854b-2f5d2aa0549a","Type":"ContainerStarted","Data":"723afee565143ce75a365b79bb2778598ef82e618acb1a5aabe06d0684adaf95"}
Jul 15 10:46:36.546579 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:36.546523    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-n2v2j" podStartSLOduration=1.5464829039999999 podStartE2EDuration="1.546482904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:36.546348183 +0000 UTC m=+65.776315713" watchObservedRunningTime="2026-07-15 10:46:36.546482904 +0000 UTC m=+65.776450437"
Jul 15 10:46:36.936938 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:36.936839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret\") pod \"global-pull-secret-syncer-5ffl8\" (UID: \"dae7d2b5-9b3e-4ffe-85a3-087ad2313052\") " pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:46:36.939794 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:36.939776    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 15 10:46:36.950183 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:36.950153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dae7d2b5-9b3e-4ffe-85a3-087ad2313052-original-pull-secret\") pod \"global-pull-secret-syncer-5ffl8\" (UID: \"dae7d2b5-9b3e-4ffe-85a3-087ad2313052\") " pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:46:36.987017 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:36.986985    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ffl8"
Jul 15 10:46:37.038269 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:37.038217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs\") pod \"network-metrics-daemon-hkvw8\" (UID: \"8a1c4669-2697-411f-80ff-c3c5819361dc\") " pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:46:37.038446 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:37.038356    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjtrg\" (UniqueName: \"kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg\") pod \"network-check-target-m7s5x\" (UID: \"11345472-e392-4a24-b60c-43b5fe6dae64\") " pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:46:37.045798 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:37.045772    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 15 10:46:37.046031 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:37.045831    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 15 10:46:37.049878 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:37.049464    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 15 10:46:37.049878 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:37.049552    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs podName:8a1c4669-2697-411f-80ff-c3c5819361dc nodeName:}" failed. No retries permitted until 2026-07-15 10:47:41.049531917 +0000 UTC m=+130.279499431 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs") pod "network-metrics-daemon-hkvw8" (UID: "8a1c4669-2697-411f-80ff-c3c5819361dc") : secret "metrics-daemon-secret" not found
Jul 15 10:46:37.051998 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:37.051833    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 15 10:46:37.070677 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:37.070650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjtrg\" (UniqueName: \"kubernetes.io/projected/11345472-e392-4a24-b60c-43b5fe6dae64-kube-api-access-tjtrg\") pod \"network-check-target-m7s5x\" (UID: \"11345472-e392-4a24-b60c-43b5fe6dae64\") " pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:46:37.088659 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:37.088574    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ww56m_6918f75f-21cc-47c8-9332-c1055bdbf5df/dns-node-resolver/0.log"
Jul 15 10:46:37.154194 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:37.154161    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5ffl8"]
Jul 15 10:46:37.159440 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:46:37.159411    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae7d2b5_9b3e_4ffe_85a3_087ad2313052.slice/crio-1db88e51bde9fb625f7edba00ea3dcafd67121e8204b9f0e0c1fc6fd9064275e WatchSource:0}: Error finding container 1db88e51bde9fb625f7edba00ea3dcafd67121e8204b9f0e0c1fc6fd9064275e: Status 404 returned error can't find the container with id 1db88e51bde9fb625f7edba00ea3dcafd67121e8204b9f0e0c1fc6fd9064275e
Jul 15 10:46:37.284279 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:37.284250    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xhf7j\""
Jul 15 10:46:37.292361 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:37.292331    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:46:37.458210 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:37.458180    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m7s5x"]
Jul 15 10:46:37.462346 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:46:37.462321    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11345472_e392_4a24_b60c_43b5fe6dae64.slice/crio-e330032fe32c6b866582976edca31f4260ed7f57b55b55420ff38bf25be89934 WatchSource:0}: Error finding container e330032fe32c6b866582976edca31f4260ed7f57b55b55420ff38bf25be89934: Status 404 returned error can't find the container with id e330032fe32c6b866582976edca31f4260ed7f57b55b55420ff38bf25be89934
Jul 15 10:46:37.524266 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:37.524227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m7s5x" event={"ID":"11345472-e392-4a24-b60c-43b5fe6dae64","Type":"ContainerStarted","Data":"e330032fe32c6b866582976edca31f4260ed7f57b55b55420ff38bf25be89934"}
Jul 15 10:46:37.525561 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:37.525530    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5ffl8" event={"ID":"dae7d2b5-9b3e-4ffe-85a3-087ad2313052","Type":"ContainerStarted","Data":"1db88e51bde9fb625f7edba00ea3dcafd67121e8204b9f0e0c1fc6fd9064275e"}
Jul 15 10:46:37.543549 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:37.543476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:46:37.543673 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:37.543607    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 15 10:46:37.543726 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:37.543673    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls podName:dfc3a194-a6c3-4cee-8f9c-225bc846d9b9 nodeName:}" failed. No retries permitted until 2026-07-15 10:47:09.543659446 +0000 UTC m=+98.773626955 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls") pod "dns-default-8f8hw" (UID: "dfc3a194-a6c3-4cee-8f9c-225bc846d9b9") : secret "dns-default-metrics-tls" not found
Jul 15 10:46:37.846872 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:37.846784    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:37.847066 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:37.846874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:37.847066 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:37.846958    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:37.847066 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:37.847024    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 15 10:46:37.847066 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:37.847047    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls podName:d30589a9-b904-4874-b34b-93d1ee5507d2 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:41.847022239 +0000 UTC m=+71.076989763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t8tvc" (UID: "d30589a9-b904-4874-b34b-93d1ee5507d2") : secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:37.847234 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:37.847082    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm podName:d30589a9-b904-4874-b34b-93d1ee5507d2 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:41.847068788 +0000 UTC m=+71.077036301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t8tvc" (UID: "d30589a9-b904-4874-b34b-93d1ee5507d2") : configmap "kube-rbac-proxy" not found
Jul 15 10:46:37.892700 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:37.892665    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hxflc_c118fdb2-4d52-4883-99c6-cf8609b6d8dd/node-ca/0.log"
Jul 15 10:46:38.301603 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:38.299757    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b5jv4_599648bd-bc11-4e5a-9664-0c4892ce9edc/serve-healthcheck-canary/0.log"
Jul 15 10:46:38.888149 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:38.888113    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-drzdz_0b55cf39-c58b-4ef8-82b1-706516c52fc9/migrator/0.log"
Jul 15 10:46:39.087998 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:39.087973    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-drzdz_0b55cf39-c58b-4ef8-82b1-706516c52fc9/graceful-termination/0.log"
Jul 15 10:46:41.885513 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:41.885451    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:41.885987 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:41.885570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:41.885987 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:41.885634    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:41.885987 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:41.885685    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 15 10:46:41.885987 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:41.885713    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls podName:d30589a9-b904-4874-b34b-93d1ee5507d2 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:49.885691279 +0000 UTC m=+79.115658809 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t8tvc" (UID: "d30589a9-b904-4874-b34b-93d1ee5507d2") : secret "insights-runtime-extractor-tls" not found
Jul 15 10:46:41.885987 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:46:41.885732    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm podName:d30589a9-b904-4874-b34b-93d1ee5507d2 nodeName:}" failed. No retries permitted until 2026-07-15 10:46:49.885721953 +0000 UTC m=+79.115689462 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t8tvc" (UID: "d30589a9-b904-4874-b34b-93d1ee5507d2") : configmap "kube-rbac-proxy" not found
Jul 15 10:46:42.541515 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:42.541472    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m7s5x" event={"ID":"11345472-e392-4a24-b60c-43b5fe6dae64","Type":"ContainerStarted","Data":"ebaa2af471d17820930413dc9c19a029a5e5353080bfd6ecb84f6f805f4e68b6"}
Jul 15 10:46:42.541672 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:42.541619    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:46:42.542937 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:42.542918    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5ffl8" event={"ID":"dae7d2b5-9b3e-4ffe-85a3-087ad2313052","Type":"ContainerStarted","Data":"916b87cd01056c64471fe791d858a202b9174f7d0cc4d4cb2738b66d8ca356b3"}
Jul 15 10:46:42.564762 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:42.564724    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m7s5x" podStartSLOduration=67.039132571 podStartE2EDuration="1m11.564711136s" totalImagesPullingTime="4.525578565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:42.564023296 +0000 UTC m=+71.793990828" watchObservedRunningTime="2026-07-15 10:46:42.564711136 +0000 UTC m=+71.794678667"
Jul 15 10:46:42.586973 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:42.586935    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5ffl8" podStartSLOduration=66.755095349 podStartE2EDuration="1m11.586924618s" totalImagesPullingTime="4.831829269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:42.586134957 +0000 UTC m=+71.816102487" watchObservedRunningTime="2026-07-15 10:46:42.586924618 +0000 UTC m=+71.816892145"
Jul 15 10:46:43.902407 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:43.902362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-m9qxt\" (UID: \"bc05cca4-8f13-4ee6-8673-a1dd0a651776\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt"
Jul 15 10:46:43.904842 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:43.904819    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc05cca4-8f13-4ee6-8673-a1dd0a651776-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-m9qxt\" (UID: \"bc05cca4-8f13-4ee6-8673-a1dd0a651776\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt"
Jul 15 10:46:44.008048 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:44.008013    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-bw2z6\""
Jul 15 10:46:44.016431 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:44.016404    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt"
Jul 15 10:46:44.151001 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:44.150967    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt"]
Jul 15 10:46:44.550122 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:44.550090    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt" event={"ID":"bc05cca4-8f13-4ee6-8673-a1dd0a651776","Type":"ContainerStarted","Data":"442d30ee2be1ddedcc523999b7ff4cfd22de3e634acea387165b051994d3aed9"}
Jul 15 10:46:46.560199 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:46.560156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt" event={"ID":"bc05cca4-8f13-4ee6-8673-a1dd0a651776","Type":"ContainerStarted","Data":"857906f9fb54d55438cc26808e5d956ea0b4bd4105744105987d4c5245358c91"}
Jul 15 10:46:46.560611 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:46.560221    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt" event={"ID":"bc05cca4-8f13-4ee6-8673-a1dd0a651776","Type":"ContainerStarted","Data":"e2958bfb2d4b7e1881f4661aeb13b71cf902269933ba9a468e75726c0ea008b8"}
Jul 15 10:46:46.580950 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:46.580909    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-m9qxt" podStartSLOduration=16.88000407 podStartE2EDuration="18.580893525s" totalImagesPullingTime="1.700889455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:46.578778437 +0000 UTC m=+75.808745968" watchObservedRunningTime="2026-07-15 10:46:46.580893525 +0000 UTC m=+75.810861055"
Jul 15 10:46:49.957906 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:49.957866    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:49.958302 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:49.957922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:49.958392 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:49.958374    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30589a9-b904-4874-b34b-93d1ee5507d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:49.960286 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:49.960264    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30589a9-b904-4874-b34b-93d1ee5507d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t8tvc\" (UID: \"d30589a9-b904-4874-b34b-93d1ee5507d2\") " pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:50.046481 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:50.046448    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t8tvc"
Jul 15 10:46:50.177214 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:50.177177    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t8tvc"]
Jul 15 10:46:50.181283 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:46:50.181257    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30589a9_b904_4874_b34b_93d1ee5507d2.slice/crio-33598fec819e3aee2e687d747dbf2a30316492152d13380f72ef4a7889d65b0b WatchSource:0}: Error finding container 33598fec819e3aee2e687d747dbf2a30316492152d13380f72ef4a7889d65b0b: Status 404 returned error can't find the container with id 33598fec819e3aee2e687d747dbf2a30316492152d13380f72ef4a7889d65b0b
Jul 15 10:46:50.574859 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:50.574824    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t8tvc" event={"ID":"d30589a9-b904-4874-b34b-93d1ee5507d2","Type":"ContainerStarted","Data":"33598fec819e3aee2e687d747dbf2a30316492152d13380f72ef4a7889d65b0b"}
Jul 15 10:46:55.592340 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:55.592293    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t8tvc" event={"ID":"d30589a9-b904-4874-b34b-93d1ee5507d2","Type":"ContainerStarted","Data":"205cb982db13a2d0b210efc3282dc53b9ea4f03524e2ab64170c7e50fa4da6d5"}
Jul 15 10:46:57.109533 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.109204    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-xftvp"]
Jul 15 10:46:57.125927 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.125891    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"]
Jul 15 10:46:57.126092 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.125991    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-xftvp"
Jul 15 10:46:57.130481 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.130449    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.138287 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.138262    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 15 10:46:57.138406 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.138262    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 15 10:46:57.141646 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.141629    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 15 10:46:57.142662 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.142635    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 15 10:46:57.142769 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.142717    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 15 10:46:57.142973 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.142952    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-xftvp"]
Jul 15 10:46:57.146179 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.145464    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-lvgqn\""
Jul 15 10:46:57.146399 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.146377    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 15 10:46:57.146542 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.146411    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 15 10:46:57.146678 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.146661    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 15 10:46:57.159702 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.159674    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"]
Jul 15 10:46:57.163287 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.163117    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 15 10:46:57.208846 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.208814    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d7778dc9b-mh9g2"]
Jul 15 10:46:57.215548 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.215521    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.215888 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.215861    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h74df\" (UniqueName: \"kubernetes.io/projected/5ca40d07-27fa-4bcb-8bc4-f01651105a6a-kube-api-access-h74df\") pod \"volume-data-source-validator-78495597f6-xftvp\" (UID: \"5ca40d07-27fa-4bcb-8bc4-f01651105a6a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-xftvp"
Jul 15 10:46:57.216011 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.215921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.216011 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.215954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.216011 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.215997    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-hub\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.216166 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.216077    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.216166 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.216113    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcbfn\" (UniqueName: \"kubernetes.io/projected/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-kube-api-access-hcbfn\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.216271 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.216163    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-ca\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.219055 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.219022    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 15 10:46:57.219254 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.219231    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 15 10:46:57.219365 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.219272    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 15 10:46:57.222243 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.222218    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mqg6r\""
Jul 15 10:46:57.242486 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.242464    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 15 10:46:57.249413 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.249380    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d7778dc9b-mh9g2"]
Jul 15 10:46:57.317517 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.317453    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcbfn\" (UniqueName: \"kubernetes.io/projected/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-kube-api-access-hcbfn\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.317679 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.317558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ffa720a-c23e-44c7-84e8-95d7724446ba-trusted-ca\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.317679 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.317601    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ffa720a-c23e-44c7-84e8-95d7724446ba-registry-tls\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.317679 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.317639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-ca\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.317847 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.317712    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjx79\" (UniqueName: \"kubernetes.io/projected/5ffa720a-c23e-44c7-84e8-95d7724446ba-kube-api-access-gjx79\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.317847 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.317747    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ffa720a-c23e-44c7-84e8-95d7724446ba-ca-trust-extracted\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.317847 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.317788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h74df\" (UniqueName: \"kubernetes.io/projected/5ca40d07-27fa-4bcb-8bc4-f01651105a6a-kube-api-access-h74df\") pod \"volume-data-source-validator-78495597f6-xftvp\" (UID: \"5ca40d07-27fa-4bcb-8bc4-f01651105a6a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-xftvp"
Jul 15 10:46:57.317847 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.317816    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ffa720a-c23e-44c7-84e8-95d7724446ba-registry-certificates\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.317847 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.317841    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ffa720a-c23e-44c7-84e8-95d7724446ba-image-registry-private-configuration\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.318663 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.317882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.318663 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.317904    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.318663 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.317928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-hub\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.318663 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.317964    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ffa720a-c23e-44c7-84e8-95d7724446ba-installation-pull-secrets\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.318663 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.317995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.318663 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.318018    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ffa720a-c23e-44c7-84e8-95d7724446ba-bound-sa-token\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.318933 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.318906    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.321248 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.321226    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.321357 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.321302    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.322858 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.322833    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-ca\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.325013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.324327    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-hub\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.332184 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.332164    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h74df\" (UniqueName: \"kubernetes.io/projected/5ca40d07-27fa-4bcb-8bc4-f01651105a6a-kube-api-access-h74df\") pod \"volume-data-source-validator-78495597f6-xftvp\" (UID: \"5ca40d07-27fa-4bcb-8bc4-f01651105a6a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-xftvp"
Jul 15 10:46:57.336294 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.336270    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcbfn\" (UniqueName: \"kubernetes.io/projected/737ddf6f-bdfe-4955-b273-2fb4ee4b127e-kube-api-access-hcbfn\") pod \"cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4\" (UID: \"737ddf6f-bdfe-4955-b273-2fb4ee4b127e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.419103 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.419006    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ffa720a-c23e-44c7-84e8-95d7724446ba-installation-pull-secrets\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.419103 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.419046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ffa720a-c23e-44c7-84e8-95d7724446ba-bound-sa-token\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.419103 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.419077    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ffa720a-c23e-44c7-84e8-95d7724446ba-trusted-ca\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.419306 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.419114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ffa720a-c23e-44c7-84e8-95d7724446ba-registry-tls\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.419306 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.419166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjx79\" (UniqueName: \"kubernetes.io/projected/5ffa720a-c23e-44c7-84e8-95d7724446ba-kube-api-access-gjx79\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.419306 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.419196    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ffa720a-c23e-44c7-84e8-95d7724446ba-ca-trust-extracted\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.419306 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.419234    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ffa720a-c23e-44c7-84e8-95d7724446ba-registry-certificates\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.419306 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.419259    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ffa720a-c23e-44c7-84e8-95d7724446ba-image-registry-private-configuration\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.420084 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.420000    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ffa720a-c23e-44c7-84e8-95d7724446ba-ca-trust-extracted\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.420326 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.420289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ffa720a-c23e-44c7-84e8-95d7724446ba-trusted-ca\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.420932 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.420910    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ffa720a-c23e-44c7-84e8-95d7724446ba-registry-certificates\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.422177 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.422146    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ffa720a-c23e-44c7-84e8-95d7724446ba-image-registry-private-configuration\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.422274 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.422251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ffa720a-c23e-44c7-84e8-95d7724446ba-registry-tls\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.422365 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.422347    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ffa720a-c23e-44c7-84e8-95d7724446ba-installation-pull-secrets\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.432274 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.432249    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ffa720a-c23e-44c7-84e8-95d7724446ba-bound-sa-token\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.433232 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.433206    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjx79\" (UniqueName: \"kubernetes.io/projected/5ffa720a-c23e-44c7-84e8-95d7724446ba-kube-api-access-gjx79\") pod \"image-registry-d7778dc9b-mh9g2\" (UID: \"5ffa720a-c23e-44c7-84e8-95d7724446ba\") " pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:57.437799 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.437768    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-xftvp"
Jul 15 10:46:57.456835 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.456817    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"
Jul 15 10:46:57.526531 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:57.526482    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:58.467410 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:58.467316    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d7778dc9b-mh9g2"]
Jul 15 10:46:58.472535 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:46:58.472487    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ffa720a_c23e_44c7_84e8_95d7724446ba.slice/crio-528dc8fc014e0d716b7783931b2dec1d88e09cc8072d6fede9213c7498bd2de1 WatchSource:0}: Error finding container 528dc8fc014e0d716b7783931b2dec1d88e09cc8072d6fede9213c7498bd2de1: Status 404 returned error can't find the container with id 528dc8fc014e0d716b7783931b2dec1d88e09cc8072d6fede9213c7498bd2de1
Jul 15 10:46:58.489010 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:58.488983    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-xftvp"]
Jul 15 10:46:58.492385 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:46:58.492361    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca40d07_27fa_4bcb_8bc4_f01651105a6a.slice/crio-ad897dc1ff1e5e974da202e98f6ee599270396ffa9fd8394eaea55ba9d630ee8 WatchSource:0}: Error finding container ad897dc1ff1e5e974da202e98f6ee599270396ffa9fd8394eaea55ba9d630ee8: Status 404 returned error can't find the container with id ad897dc1ff1e5e974da202e98f6ee599270396ffa9fd8394eaea55ba9d630ee8
Jul 15 10:46:58.502360 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:58.502334    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4"]
Jul 15 10:46:58.517551 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:46:58.517526    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod737ddf6f_bdfe_4955_b273_2fb4ee4b127e.slice/crio-95ea7269933d213d4ccdbceab6a4de848879b276fe845664b697ef0e15f7f2ea WatchSource:0}: Error finding container 95ea7269933d213d4ccdbceab6a4de848879b276fe845664b697ef0e15f7f2ea: Status 404 returned error can't find the container with id 95ea7269933d213d4ccdbceab6a4de848879b276fe845664b697ef0e15f7f2ea
Jul 15 10:46:58.602901 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:58.602821    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2" event={"ID":"5ffa720a-c23e-44c7-84e8-95d7724446ba","Type":"ContainerStarted","Data":"883f5721c511107ed64f4f7039754b0e07b03f597cfb1aff7889f009ce8812f1"}
Jul 15 10:46:58.602901 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:58.602873    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2" event={"ID":"5ffa720a-c23e-44c7-84e8-95d7724446ba","Type":"ContainerStarted","Data":"528dc8fc014e0d716b7783931b2dec1d88e09cc8072d6fede9213c7498bd2de1"}
Jul 15 10:46:58.602901 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:58.602900    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:46:58.604619 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:58.604595    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t8tvc" event={"ID":"d30589a9-b904-4874-b34b-93d1ee5507d2","Type":"ContainerStarted","Data":"2589697e0ec1b8dc5d7cbdc69343d402255fb136dbe3905f1d2c97eff2474298"}
Jul 15 10:46:58.605733 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:58.605711    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-xftvp" event={"ID":"5ca40d07-27fa-4bcb-8bc4-f01651105a6a","Type":"ContainerStarted","Data":"ad897dc1ff1e5e974da202e98f6ee599270396ffa9fd8394eaea55ba9d630ee8"}
Jul 15 10:46:58.606835 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:58.606812    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4" event={"ID":"737ddf6f-bdfe-4955-b273-2fb4ee4b127e","Type":"ContainerStarted","Data":"95ea7269933d213d4ccdbceab6a4de848879b276fe845664b697ef0e15f7f2ea"}
Jul 15 10:46:58.623326 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:46:58.623282    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2" podStartSLOduration=1.623268248 podStartE2EDuration="1.623268248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:46:58.623066567 +0000 UTC m=+87.853034099" watchObservedRunningTime="2026-07-15 10:46:58.623268248 +0000 UTC m=+87.853235757"
Jul 15 10:47:00.621974 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:00.621935    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-xftvp" event={"ID":"5ca40d07-27fa-4bcb-8bc4-f01651105a6a","Type":"ContainerStarted","Data":"d4d6a4869f73c65d4c290dd6a8a87d6d7777f3bd2aacde17bc9f601209eb92d7"}
Jul 15 10:47:00.640985 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:00.640941    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-xftvp" podStartSLOduration=2.493572622 podStartE2EDuration="3.640927798s" totalImagesPullingTime="1.147355176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:00.639850272 +0000 UTC m=+89.869817806" watchObservedRunningTime="2026-07-15 10:47:00.640927798 +0000 UTC m=+89.870895328"
Jul 15 10:47:04.638783 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:04.638745    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t8tvc" event={"ID":"d30589a9-b904-4874-b34b-93d1ee5507d2","Type":"ContainerStarted","Data":"b5230267d195d292c33fbb8fd2534ac764b3a315f8a1e82265b11fec9bfa4327"}
Jul 15 10:47:04.640742 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:04.640715    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4" event={"ID":"737ddf6f-bdfe-4955-b273-2fb4ee4b127e","Type":"ContainerStarted","Data":"61c7cde6b28038014c8d49db728953b6ed6a5f3215058079dcc99fab6dea5cd8"}
Jul 15 10:47:04.640861 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:04.640751    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4" event={"ID":"737ddf6f-bdfe-4955-b273-2fb4ee4b127e","Type":"ContainerStarted","Data":"2ba94700b58ba8dbd596e336b62efd92cf60d37a59491bcf352cd6720294256b"}
Jul 15 10:47:04.640861 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:04.640782    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4" event={"ID":"737ddf6f-bdfe-4955-b273-2fb4ee4b127e","Type":"ContainerStarted","Data":"5dbed9721deca9bc5ddf0262d23cc4cecefc6a31ff9b372c6b5fc918e342fba7"}
Jul 15 10:47:04.666244 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:04.666182    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-t8tvc" podStartSLOduration=17.20658891 podStartE2EDuration="30.66616646s" totalImagesPullingTime="13.45957755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 10:46:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:04.665147616 +0000 UTC m=+93.895115140" watchObservedRunningTime="2026-07-15 10:47:04.66616646 +0000 UTC m=+93.896133992"
Jul 15 10:47:04.691351 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:04.691306    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b7f7f4bdc-bl9c4" podStartSLOduration=2.396379731 podStartE2EDuration="7.691293496s" totalImagesPullingTime="5.294913765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:04.690800927 +0000 UTC m=+93.920768457" watchObservedRunningTime="2026-07-15 10:47:04.691293496 +0000 UTC m=+93.921261027"
Jul 15 10:47:09.627309 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:09.627196    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:47:09.629725 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:09.629706    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfc3a194-a6c3-4cee-8f9c-225bc846d9b9-metrics-tls\") pod \"dns-default-8f8hw\" (UID: \"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9\") " pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:47:09.858015 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:09.857977    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8cmqw\""
Jul 15 10:47:09.865305 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:09.865280    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:47:10.006932 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.006882    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8f8hw"]
Jul 15 10:47:10.483582 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.483542    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jj947"]
Jul 15 10:47:10.489038 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.489019    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.493786 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.493758    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 15 10:47:10.494285 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.494114    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 15 10:47:10.494868 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.494442    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 15 10:47:10.494868 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.494475    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 15 10:47:10.494868 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.494654    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 15 10:47:10.494868 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.494715    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q7hjh\""
Jul 15 10:47:10.495102 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.494877    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 15 10:47:10.534835 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.534796    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-metrics-client-ca\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.535029 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.534848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-root\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.535029 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.534890    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-sys\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.535029 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.534921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-node-exporter-accelerators-collector-config\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.535029 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.534966    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-node-exporter-wtmp\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.535236 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.535041    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-node-exporter-textfile\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.535236 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.535081    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-node-exporter-tls\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.535236 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.535110    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjgbt\" (UniqueName: \"kubernetes.io/projected/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-kube-api-access-bjgbt\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.535236 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.535187    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.635613 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.635583    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-root\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.636044 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.635627    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-sys\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.636044 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.635667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-node-exporter-accelerators-collector-config\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.636044 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.635696    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-node-exporter-wtmp\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.636044 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.635710    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-root\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.636044 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.635759    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-node-exporter-textfile\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.636044 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.635770    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-sys\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.636044 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.635793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-node-exporter-tls\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.636044 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.635846    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjgbt\" (UniqueName: \"kubernetes.io/projected/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-kube-api-access-bjgbt\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.636044 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.635848    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-node-exporter-wtmp\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.636044 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.635887    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.636044 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.635918    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-metrics-client-ca\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.636563 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.636187    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-node-exporter-textfile\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.636563 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.636422    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-node-exporter-accelerators-collector-config\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.636563 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.636520    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-metrics-client-ca\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.638566 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.638540    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.638674 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.638606    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-node-exporter-tls\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.652059 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.651289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjgbt\" (UniqueName: \"kubernetes.io/projected/cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4-kube-api-access-bjgbt\") pod \"node-exporter-jj947\" (UID: \"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4\") " pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.662359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.662329    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8f8hw" event={"ID":"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9","Type":"ContainerStarted","Data":"c547f783153831f28e48ef46c311930e700435f50585749b20d63ef26a05b410"}
Jul 15 10:47:10.799939 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:10.799908    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jj947"
Jul 15 10:47:10.811674 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:47:10.811640    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1d0e6f_d42d_401a_8f10_8f6eac3ba3c4.slice/crio-ddaadf10c82d2425986b6719e4c6ac80e07dd01db70ab2065d1063b8344550dd WatchSource:0}: Error finding container ddaadf10c82d2425986b6719e4c6ac80e07dd01db70ab2065d1063b8344550dd: Status 404 returned error can't find the container with id ddaadf10c82d2425986b6719e4c6ac80e07dd01db70ab2065d1063b8344550dd
Jul 15 10:47:11.667201 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:11.667164    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jj947" event={"ID":"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4","Type":"ContainerStarted","Data":"ddaadf10c82d2425986b6719e4c6ac80e07dd01db70ab2065d1063b8344550dd"}
Jul 15 10:47:12.673536 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:12.672799    2566 generic.go:356] "Generic (PLEG): container finished" podID="cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4" containerID="b2d8f18ba9500d5c598abd836f8a4a62c7309ed185bccde152b425e5af631c6d" exitCode=0
Jul 15 10:47:12.673536 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:12.672890    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jj947" event={"ID":"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4","Type":"ContainerDied","Data":"b2d8f18ba9500d5c598abd836f8a4a62c7309ed185bccde152b425e5af631c6d"}
Jul 15 10:47:13.548913 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:13.548884    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m7s5x"
Jul 15 10:47:13.678898 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:13.678864    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8f8hw" event={"ID":"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9","Type":"ContainerStarted","Data":"e5dd76296b31757f1b2a86bc564d7b3b40a56a8c385a28ec4ba7a4e31c07f603"}
Jul 15 10:47:13.679319 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:13.678958    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:47:13.679319 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:13.678989    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8f8hw" event={"ID":"dfc3a194-a6c3-4cee-8f9c-225bc846d9b9","Type":"ContainerStarted","Data":"911c973154fe6a0f39932f9e8f6d75f2388db6c545c119051d1f6fe4834b0c0d"}
Jul 15 10:47:13.680872 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:13.680850    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jj947" event={"ID":"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4","Type":"ContainerStarted","Data":"a6d334e6db52ed7afb2e479e9492a349a738f13873788874712bdb7cbfc90a00"}
Jul 15 10:47:13.680995 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:13.680889    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jj947" event={"ID":"cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4","Type":"ContainerStarted","Data":"9c5a3af79defca3a43b3995ed00538e1b58797319953cd1aab2699cc16154873"}
Jul 15 10:47:13.738184 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:13.738138    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8f8hw" podStartSLOduration=65.571209255 podStartE2EDuration="1m8.738117705s" totalImagesPullingTime="3.16690845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:13.734775622 +0000 UTC m=+102.964743168" watchObservedRunningTime="2026-07-15 10:47:13.738117705 +0000 UTC m=+102.968085237"
Jul 15 10:47:19.616401 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:19.616373    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d7778dc9b-mh9g2"
Jul 15 10:47:19.640095 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:19.640045    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jj947" podStartSLOduration=8.770475219 podStartE2EDuration="9.640029175s" totalImagesPullingTime="798.870344ms" totalInitContainerRuntime="70.683612ms" isStatefulPod=true podCreationTimestamp="2026-07-15 10:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:13.793764839 +0000 UTC m=+103.023732370" watchObservedRunningTime="2026-07-15 10:47:19.640029175 +0000 UTC m=+108.869996719"
Jul 15 10:47:23.687076 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:23.687048    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8f8hw"
Jul 15 10:47:41.149048 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:41.149010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs\") pod \"network-metrics-daemon-hkvw8\" (UID: \"8a1c4669-2697-411f-80ff-c3c5819361dc\") " pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:47:41.151416 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:41.151387    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a1c4669-2697-411f-80ff-c3c5819361dc-metrics-certs\") pod \"network-metrics-daemon-hkvw8\" (UID: \"8a1c4669-2697-411f-80ff-c3c5819361dc\") " pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:47:41.374020 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:41.373991    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vfsm8\""
Jul 15 10:47:41.382237 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:41.382204    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hkvw8"
Jul 15 10:47:41.544008 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:41.543980    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hkvw8"]
Jul 15 10:47:41.547720 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:47:41.547693    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1c4669_2697_411f_80ff_c3c5819361dc.slice/crio-c183c589d2044fd338e3cf03d118c8d146385ce27d3c7d525b13c27f671dfb44 WatchSource:0}: Error finding container c183c589d2044fd338e3cf03d118c8d146385ce27d3c7d525b13c27f671dfb44: Status 404 returned error can't find the container with id c183c589d2044fd338e3cf03d118c8d146385ce27d3c7d525b13c27f671dfb44
Jul 15 10:47:41.780149 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:41.780113    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hkvw8" event={"ID":"8a1c4669-2697-411f-80ff-c3c5819361dc","Type":"ContainerStarted","Data":"c183c589d2044fd338e3cf03d118c8d146385ce27d3c7d525b13c27f671dfb44"}
Jul 15 10:47:42.785356 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:42.785322    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hkvw8" event={"ID":"8a1c4669-2697-411f-80ff-c3c5819361dc","Type":"ContainerStarted","Data":"b758cdc5f69de41741f945c0915e36159106420958ecb02d2788ba0985d8271e"}
Jul 15 10:47:42.785735 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:42.785404    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hkvw8" event={"ID":"8a1c4669-2697-411f-80ff-c3c5819361dc","Type":"ContainerStarted","Data":"7d0ee42a1cddfb6add140ffcd58960bb81fdd9edde99a7beeda7cb284d16e796"}
Jul 15 10:47:42.804633 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:42.804588    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hkvw8" podStartSLOduration=130.932654058 podStartE2EDuration="2m11.804574987s" totalImagesPullingTime="871.920929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:47:42.803709676 +0000 UTC m=+132.033677207" watchObservedRunningTime="2026-07-15 10:47:42.804574987 +0000 UTC m=+132.034542518"
Jul 15 10:47:56.832328 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:56.832303    2566 generic.go:356] "Generic (PLEG): container finished" podID="2df82869-758b-450d-bec7-14b4e125ec49" containerID="7f58bf9e46e2fd1b6db14cf79cb0d23326bdfe5d134aabb483f2302b3acc602f" exitCode=0
Jul 15 10:47:56.832680 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:56.832377    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw" event={"ID":"2df82869-758b-450d-bec7-14b4e125ec49","Type":"ContainerDied","Data":"7f58bf9e46e2fd1b6db14cf79cb0d23326bdfe5d134aabb483f2302b3acc602f"}
Jul 15 10:47:56.832720 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:56.832705    2566 scope.go:117] "RemoveContainer" containerID="7f58bf9e46e2fd1b6db14cf79cb0d23326bdfe5d134aabb483f2302b3acc602f"
Jul 15 10:47:57.837834 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:47:57.837793    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-r7qfw" event={"ID":"2df82869-758b-450d-bec7-14b4e125ec49","Type":"ContainerStarted","Data":"d76428799854a47a684467321291755ce33dc01617f1a7c9cf310c37ba2829e2"}
Jul 15 10:50:31.134753 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:50:31.134722    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 10:50:31.135252 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:50:31.135014    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 10:50:31.146779 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:50:31.146757    2566 kubelet.go:1648] "Image garbage collection succeeded"
Jul 15 10:52:52.213328 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.211353    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-749567458d-svt6l"]
Jul 15 10:52:52.216208 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.216185    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-749567458d-svt6l"
Jul 15 10:52:52.220435 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.220413    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 15 10:52:52.220928 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.220906    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-skxlp\""
Jul 15 10:52:52.221508 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.221477    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 15 10:52:52.221598 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.221566    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 15 10:52:52.230210 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.230181    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-749567458d-svt6l"]
Jul 15 10:52:52.281084 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.281057    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-gzqrl"]
Jul 15 10:52:52.284153 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.284136    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-gzqrl"
Jul 15 10:52:52.286487 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.286445    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3285dfeb-997a-409b-afac-43462bdb37f1-cert\") pod \"kserve-controller-manager-749567458d-svt6l\" (UID: \"3285dfeb-997a-409b-afac-43462bdb37f1\") " pod="kserve/kserve-controller-manager-749567458d-svt6l"
Jul 15 10:52:52.286625 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.286612    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29vhj\" (UniqueName: \"kubernetes.io/projected/3285dfeb-997a-409b-afac-43462bdb37f1-kube-api-access-29vhj\") pod \"kserve-controller-manager-749567458d-svt6l\" (UID: \"3285dfeb-997a-409b-afac-43462bdb37f1\") " pod="kserve/kserve-controller-manager-749567458d-svt6l"
Jul 15 10:52:52.287379 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.287357    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-kfrtf\""
Jul 15 10:52:52.287475 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.287407    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 15 10:52:52.298120 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.298095    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-gzqrl"]
Jul 15 10:52:52.387235 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.387200    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2309ec5-89b9-40d9-9521-f67f88f4c4a1-data\") pod \"seaweedfs-64568bcd49-gzqrl\" (UID: \"e2309ec5-89b9-40d9-9521-f67f88f4c4a1\") " pod="kserve/seaweedfs-64568bcd49-gzqrl"
Jul 15 10:52:52.387408 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.387244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29vhj\" (UniqueName: \"kubernetes.io/projected/3285dfeb-997a-409b-afac-43462bdb37f1-kube-api-access-29vhj\") pod \"kserve-controller-manager-749567458d-svt6l\" (UID: \"3285dfeb-997a-409b-afac-43462bdb37f1\") " pod="kserve/kserve-controller-manager-749567458d-svt6l"
Jul 15 10:52:52.387408 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.387318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3285dfeb-997a-409b-afac-43462bdb37f1-cert\") pod \"kserve-controller-manager-749567458d-svt6l\" (UID: \"3285dfeb-997a-409b-afac-43462bdb37f1\") " pod="kserve/kserve-controller-manager-749567458d-svt6l"
Jul 15 10:52:52.387408 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.387365    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h66wr\" (UniqueName: \"kubernetes.io/projected/e2309ec5-89b9-40d9-9521-f67f88f4c4a1-kube-api-access-h66wr\") pod \"seaweedfs-64568bcd49-gzqrl\" (UID: \"e2309ec5-89b9-40d9-9521-f67f88f4c4a1\") " pod="kserve/seaweedfs-64568bcd49-gzqrl"
Jul 15 10:52:52.387606 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:52:52.387480    2566 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 15 10:52:52.387606 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:52:52.387567    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3285dfeb-997a-409b-afac-43462bdb37f1-cert podName:3285dfeb-997a-409b-afac-43462bdb37f1 nodeName:}" failed. No retries permitted until 2026-07-15 10:52:52.887546961 +0000 UTC m=+442.117514480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3285dfeb-997a-409b-afac-43462bdb37f1-cert") pod "kserve-controller-manager-749567458d-svt6l" (UID: "3285dfeb-997a-409b-afac-43462bdb37f1") : secret "kserve-webhook-server-cert" not found
Jul 15 10:52:52.396800 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.396779    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29vhj\" (UniqueName: \"kubernetes.io/projected/3285dfeb-997a-409b-afac-43462bdb37f1-kube-api-access-29vhj\") pod \"kserve-controller-manager-749567458d-svt6l\" (UID: \"3285dfeb-997a-409b-afac-43462bdb37f1\") " pod="kserve/kserve-controller-manager-749567458d-svt6l"
Jul 15 10:52:52.488251 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.488180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h66wr\" (UniqueName: \"kubernetes.io/projected/e2309ec5-89b9-40d9-9521-f67f88f4c4a1-kube-api-access-h66wr\") pod \"seaweedfs-64568bcd49-gzqrl\" (UID: \"e2309ec5-89b9-40d9-9521-f67f88f4c4a1\") " pod="kserve/seaweedfs-64568bcd49-gzqrl"
Jul 15 10:52:52.488251 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.488226    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2309ec5-89b9-40d9-9521-f67f88f4c4a1-data\") pod \"seaweedfs-64568bcd49-gzqrl\" (UID: \"e2309ec5-89b9-40d9-9521-f67f88f4c4a1\") " pod="kserve/seaweedfs-64568bcd49-gzqrl"
Jul 15 10:52:52.488570 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.488554    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2309ec5-89b9-40d9-9521-f67f88f4c4a1-data\") pod \"seaweedfs-64568bcd49-gzqrl\" (UID: \"e2309ec5-89b9-40d9-9521-f67f88f4c4a1\") " pod="kserve/seaweedfs-64568bcd49-gzqrl"
Jul 15 10:52:52.498189 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.498161    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h66wr\" (UniqueName: \"kubernetes.io/projected/e2309ec5-89b9-40d9-9521-f67f88f4c4a1-kube-api-access-h66wr\") pod \"seaweedfs-64568bcd49-gzqrl\" (UID: \"e2309ec5-89b9-40d9-9521-f67f88f4c4a1\") " pod="kserve/seaweedfs-64568bcd49-gzqrl"
Jul 15 10:52:52.594810 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.594781    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-gzqrl"
Jul 15 10:52:52.721158 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.721070    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-gzqrl"]
Jul 15 10:52:52.723824 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:52:52.723796    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2309ec5_89b9_40d9_9521_f67f88f4c4a1.slice/crio-b93cb8321e022a270e9550e789163b9d3706a328ed379f79b9b0d519f3cb5e96 WatchSource:0}: Error finding container b93cb8321e022a270e9550e789163b9d3706a328ed379f79b9b0d519f3cb5e96: Status 404 returned error can't find the container with id b93cb8321e022a270e9550e789163b9d3706a328ed379f79b9b0d519f3cb5e96
Jul 15 10:52:52.725396 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.725382    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 10:52:52.824880 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.824848    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-gzqrl" event={"ID":"e2309ec5-89b9-40d9-9521-f67f88f4c4a1","Type":"ContainerStarted","Data":"b93cb8321e022a270e9550e789163b9d3706a328ed379f79b9b0d519f3cb5e96"}
Jul 15 10:52:52.891120 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.891091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3285dfeb-997a-409b-afac-43462bdb37f1-cert\") pod \"kserve-controller-manager-749567458d-svt6l\" (UID: \"3285dfeb-997a-409b-afac-43462bdb37f1\") " pod="kserve/kserve-controller-manager-749567458d-svt6l"
Jul 15 10:52:52.893553 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:52.893531    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3285dfeb-997a-409b-afac-43462bdb37f1-cert\") pod \"kserve-controller-manager-749567458d-svt6l\" (UID: \"3285dfeb-997a-409b-afac-43462bdb37f1\") " pod="kserve/kserve-controller-manager-749567458d-svt6l"
Jul 15 10:52:53.127968 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:53.127891    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-749567458d-svt6l"
Jul 15 10:52:53.251898 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:53.251869    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-749567458d-svt6l"]
Jul 15 10:52:53.254891 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:52:53.254864    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3285dfeb_997a_409b_afac_43462bdb37f1.slice/crio-7e23aaaf5bb2afe2893fd06edab3551ff17932dd5f881453c14da6dda70a1178 WatchSource:0}: Error finding container 7e23aaaf5bb2afe2893fd06edab3551ff17932dd5f881453c14da6dda70a1178: Status 404 returned error can't find the container with id 7e23aaaf5bb2afe2893fd06edab3551ff17932dd5f881453c14da6dda70a1178
Jul 15 10:52:53.831053 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:53.831011    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-749567458d-svt6l" event={"ID":"3285dfeb-997a-409b-afac-43462bdb37f1","Type":"ContainerStarted","Data":"7e23aaaf5bb2afe2893fd06edab3551ff17932dd5f881453c14da6dda70a1178"}
Jul 15 10:52:57.849603 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:57.849564    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-749567458d-svt6l" event={"ID":"3285dfeb-997a-409b-afac-43462bdb37f1","Type":"ContainerStarted","Data":"935b133a0b073803d3fc97fb48e0e2f06fd0a2123e23bbc7671a1d6e2baab87f"}
Jul 15 10:52:57.850031 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:57.849704    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-749567458d-svt6l"
Jul 15 10:52:57.850930 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:57.850910    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-gzqrl" event={"ID":"e2309ec5-89b9-40d9-9521-f67f88f4c4a1","Type":"ContainerStarted","Data":"6db075cadf8b83b6e7545f983392e789e101df471bbffa507bb4e4f87e93882d"}
Jul 15 10:52:57.851057 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:57.851041    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-gzqrl"
Jul 15 10:52:57.867103 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:57.867064    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-749567458d-svt6l" podStartSLOduration=2.3426597080000002 podStartE2EDuration="5.867052152s" totalImagesPullingTime="3.524392444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:52:57.865872381 +0000 UTC m=+447.095839912" watchObservedRunningTime="2026-07-15 10:52:57.867052152 +0000 UTC m=+447.097019698"
Jul 15 10:52:57.881517 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:52:57.881454    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-gzqrl" podStartSLOduration=1.774901018 podStartE2EDuration="5.881438393s" totalImagesPullingTime="4.106537375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:52:57.881050122 +0000 UTC m=+447.111017653" watchObservedRunningTime="2026-07-15 10:52:57.881438393 +0000 UTC m=+447.111405931"
Jul 15 10:53:03.857184 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:03.857154    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-gzqrl"
Jul 15 10:53:28.860852 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:28.860817    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-749567458d-svt6l"
Jul 15 10:53:47.768176 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:47.768138    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dg8bp"]
Jul 15 10:53:47.771435 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:47.771413    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dg8bp"
Jul 15 10:53:47.774889 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:47.774868    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 15 10:53:47.775016 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:47.774868    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 15 10:53:47.775259 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:47.775242    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-pzntr\""
Jul 15 10:53:47.785171 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:47.785147    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dg8bp"]
Jul 15 10:53:47.797832 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:47.797809    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03f391dc-6b96-4c86-b26f-75d43311d0e8-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-dg8bp\" (UID: \"03f391dc-6b96-4c86-b26f-75d43311d0e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dg8bp"
Jul 15 10:53:47.797921 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:47.797865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tds5f\" (UniqueName: \"kubernetes.io/projected/03f391dc-6b96-4c86-b26f-75d43311d0e8-kube-api-access-tds5f\") pod \"cert-manager-operator-controller-manager-684f79c9b9-dg8bp\" (UID: \"03f391dc-6b96-4c86-b26f-75d43311d0e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dg8bp"
Jul 15 10:53:47.899128 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:47.899090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tds5f\" (UniqueName: \"kubernetes.io/projected/03f391dc-6b96-4c86-b26f-75d43311d0e8-kube-api-access-tds5f\") pod \"cert-manager-operator-controller-manager-684f79c9b9-dg8bp\" (UID: \"03f391dc-6b96-4c86-b26f-75d43311d0e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dg8bp"
Jul 15 10:53:47.899298 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:47.899138    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03f391dc-6b96-4c86-b26f-75d43311d0e8-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-dg8bp\" (UID: \"03f391dc-6b96-4c86-b26f-75d43311d0e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dg8bp"
Jul 15 10:53:47.899482 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:47.899466    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03f391dc-6b96-4c86-b26f-75d43311d0e8-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-dg8bp\" (UID: \"03f391dc-6b96-4c86-b26f-75d43311d0e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dg8bp"
Jul 15 10:53:47.919028 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:47.918999    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tds5f\" (UniqueName: \"kubernetes.io/projected/03f391dc-6b96-4c86-b26f-75d43311d0e8-kube-api-access-tds5f\") pod \"cert-manager-operator-controller-manager-684f79c9b9-dg8bp\" (UID: \"03f391dc-6b96-4c86-b26f-75d43311d0e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dg8bp"
Jul 15 10:53:48.080624 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:48.080548    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dg8bp"
Jul 15 10:53:48.233397 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:48.233367    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dg8bp"]
Jul 15 10:53:48.236991 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:53:48.236962    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f391dc_6b96_4c86_b26f_75d43311d0e8.slice/crio-73c3844c8da8c16e1c6127ea13b0790cd04b0e8eb996a01626b5aa9ef16c3192 WatchSource:0}: Error finding container 73c3844c8da8c16e1c6127ea13b0790cd04b0e8eb996a01626b5aa9ef16c3192: Status 404 returned error can't find the container with id 73c3844c8da8c16e1c6127ea13b0790cd04b0e8eb996a01626b5aa9ef16c3192
Jul 15 10:53:49.043514 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:49.043466    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dg8bp" event={"ID":"03f391dc-6b96-4c86-b26f-75d43311d0e8","Type":"ContainerStarted","Data":"73c3844c8da8c16e1c6127ea13b0790cd04b0e8eb996a01626b5aa9ef16c3192"}
Jul 15 10:53:52.056853 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:52.056818    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dg8bp" event={"ID":"03f391dc-6b96-4c86-b26f-75d43311d0e8","Type":"ContainerStarted","Data":"d5fb7ab170f2853b6d44ae5dcb9ff4a77788613c13cb1b3deb1b700db37b5965"}
Jul 15 10:53:52.080253 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:53:52.080208    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dg8bp" podStartSLOduration=2.257739978 podStartE2EDuration="5.08019302s" totalImagesPullingTime="2.822453042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:53:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:53:52.079051553 +0000 UTC m=+501.309019272" watchObservedRunningTime="2026-07-15 10:53:52.08019302 +0000 UTC m=+501.310160551"
Jul 15 10:54:03.337075 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:03.337035    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s9h4l"]
Jul 15 10:54:03.340088 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:03.340070    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-s9h4l"
Jul 15 10:54:03.342711 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:03.342685    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 15 10:54:03.342843 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:03.342715    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 15 10:54:03.342843 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:03.342757    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-smgbz\""
Jul 15 10:54:03.350197 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:03.350170    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s9h4l"]
Jul 15 10:54:03.418721 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:03.418683    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsl7x\" (UniqueName: \"kubernetes.io/projected/ac1924ac-6c05-4639-9c14-596bbe1f7b13-kube-api-access-xsl7x\") pod \"cert-manager-55b4d954d5-s9h4l\" (UID: \"ac1924ac-6c05-4639-9c14-596bbe1f7b13\") " pod="cert-manager/cert-manager-55b4d954d5-s9h4l"
Jul 15 10:54:03.418901 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:03.418731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac1924ac-6c05-4639-9c14-596bbe1f7b13-bound-sa-token\") pod \"cert-manager-55b4d954d5-s9h4l\" (UID: \"ac1924ac-6c05-4639-9c14-596bbe1f7b13\") " pod="cert-manager/cert-manager-55b4d954d5-s9h4l"
Jul 15 10:54:03.519796 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:03.519753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsl7x\" (UniqueName: \"kubernetes.io/projected/ac1924ac-6c05-4639-9c14-596bbe1f7b13-kube-api-access-xsl7x\") pod \"cert-manager-55b4d954d5-s9h4l\" (UID: \"ac1924ac-6c05-4639-9c14-596bbe1f7b13\") " pod="cert-manager/cert-manager-55b4d954d5-s9h4l"
Jul 15 10:54:03.519796 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:03.519804    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac1924ac-6c05-4639-9c14-596bbe1f7b13-bound-sa-token\") pod \"cert-manager-55b4d954d5-s9h4l\" (UID: \"ac1924ac-6c05-4639-9c14-596bbe1f7b13\") " pod="cert-manager/cert-manager-55b4d954d5-s9h4l"
Jul 15 10:54:03.531203 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:03.531168    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac1924ac-6c05-4639-9c14-596bbe1f7b13-bound-sa-token\") pod \"cert-manager-55b4d954d5-s9h4l\" (UID: \"ac1924ac-6c05-4639-9c14-596bbe1f7b13\") " pod="cert-manager/cert-manager-55b4d954d5-s9h4l"
Jul 15 10:54:03.531311 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:03.531272    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsl7x\" (UniqueName: \"kubernetes.io/projected/ac1924ac-6c05-4639-9c14-596bbe1f7b13-kube-api-access-xsl7x\") pod \"cert-manager-55b4d954d5-s9h4l\" (UID: \"ac1924ac-6c05-4639-9c14-596bbe1f7b13\") " pod="cert-manager/cert-manager-55b4d954d5-s9h4l"
Jul 15 10:54:03.649850 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:03.649769    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-s9h4l"
Jul 15 10:54:03.777669 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:03.777636    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s9h4l"]
Jul 15 10:54:03.780904 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:54:03.780873    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac1924ac_6c05_4639_9c14_596bbe1f7b13.slice/crio-732db9c40c93bd4cb699aef33ee79e350d47ae83836b48a26936c47c34f93a6d WatchSource:0}: Error finding container 732db9c40c93bd4cb699aef33ee79e350d47ae83836b48a26936c47c34f93a6d: Status 404 returned error can't find the container with id 732db9c40c93bd4cb699aef33ee79e350d47ae83836b48a26936c47c34f93a6d
Jul 15 10:54:04.105137 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:04.105102    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-s9h4l" event={"ID":"ac1924ac-6c05-4639-9c14-596bbe1f7b13","Type":"ContainerStarted","Data":"732db9c40c93bd4cb699aef33ee79e350d47ae83836b48a26936c47c34f93a6d"}
Jul 15 10:54:07.120013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:07.119979    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-s9h4l" event={"ID":"ac1924ac-6c05-4639-9c14-596bbe1f7b13","Type":"ContainerStarted","Data":"5bb843453e0cf8924feb73922a5ab6c3610c6a61159515d11f19121dbef63cee"}
Jul 15 10:54:07.139724 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:07.139670    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-s9h4l" podStartSLOduration=1.573194478 podStartE2EDuration="4.139655237s" totalImagesPullingTime="2.566460759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:54:07.137772414 +0000 UTC m=+516.367739946" watchObservedRunningTime="2026-07-15 10:54:07.139655237 +0000 UTC m=+516.369622768"
Jul 15 10:54:26.478443 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.478365    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"]
Jul 15 10:54:26.484716 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.484695    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.488625 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.488602    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 15 10:54:26.488625 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.488623    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 15 10:54:26.488807 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.488623    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 15 10:54:26.488807 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.488685    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-skcld\""
Jul 15 10:54:26.495320 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.495297    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"]
Jul 15 10:54:26.593600 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.593554    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63aad8c8-2687-4adf-9bab-9c36b0086419-workload-certs\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.593774 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.593607    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63aad8c8-2687-4adf-9bab-9c36b0086419-istio-token\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.593774 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.593658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tlkv\" (UniqueName: \"kubernetes.io/projected/63aad8c8-2687-4adf-9bab-9c36b0086419-kube-api-access-8tlkv\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.593774 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.593699    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63aad8c8-2687-4adf-9bab-9c36b0086419-workload-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.593774 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.593740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63aad8c8-2687-4adf-9bab-9c36b0086419-istio-data\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.593774 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.593762    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63aad8c8-2687-4adf-9bab-9c36b0086419-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.594013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.593826    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63aad8c8-2687-4adf-9bab-9c36b0086419-credential-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.594013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.593856    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63aad8c8-2687-4adf-9bab-9c36b0086419-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.594013 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.593886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63aad8c8-2687-4adf-9bab-9c36b0086419-istio-envoy\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.695231 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.695199    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63aad8c8-2687-4adf-9bab-9c36b0086419-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.695231 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.695230    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63aad8c8-2687-4adf-9bab-9c36b0086419-istio-envoy\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.695460 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.695257    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63aad8c8-2687-4adf-9bab-9c36b0086419-workload-certs\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.695460 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.695278    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63aad8c8-2687-4adf-9bab-9c36b0086419-istio-token\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.695460 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.695293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tlkv\" (UniqueName: \"kubernetes.io/projected/63aad8c8-2687-4adf-9bab-9c36b0086419-kube-api-access-8tlkv\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.695460 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.695314    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63aad8c8-2687-4adf-9bab-9c36b0086419-workload-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.695460 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.695350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63aad8c8-2687-4adf-9bab-9c36b0086419-istio-data\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.695740 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.695586    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63aad8c8-2687-4adf-9bab-9c36b0086419-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.695740 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.695645    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63aad8c8-2687-4adf-9bab-9c36b0086419-credential-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.695844 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.695738    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63aad8c8-2687-4adf-9bab-9c36b0086419-workload-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.695844 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.695791    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63aad8c8-2687-4adf-9bab-9c36b0086419-istio-data\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.695932 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.695855    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63aad8c8-2687-4adf-9bab-9c36b0086419-workload-certs\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.695993 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.695975    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63aad8c8-2687-4adf-9bab-9c36b0086419-credential-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.696216 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.696022    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63aad8c8-2687-4adf-9bab-9c36b0086419-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.697721 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.697694    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63aad8c8-2687-4adf-9bab-9c36b0086419-istio-envoy\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.698039 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.698023    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63aad8c8-2687-4adf-9bab-9c36b0086419-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.704083 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.704060    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63aad8c8-2687-4adf-9bab-9c36b0086419-istio-token\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.704188 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.704121    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tlkv\" (UniqueName: \"kubernetes.io/projected/63aad8c8-2687-4adf-9bab-9c36b0086419-kube-api-access-8tlkv\") pod \"openshift-ai-inference-openshift-default-866675cbb8-fmhhm\" (UID: \"63aad8c8-2687-4adf-9bab-9c36b0086419\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.797141 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.797097    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:26.932556 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:26.932523    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"]
Jul 15 10:54:26.935458 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:54:26.935429    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63aad8c8_2687_4adf_9bab_9c36b0086419.slice/crio-6934fc522cd1a43555d88aa1baa80858c6dc99c783324c0be99edde7f1679f22 WatchSource:0}: Error finding container 6934fc522cd1a43555d88aa1baa80858c6dc99c783324c0be99edde7f1679f22: Status 404 returned error can't find the container with id 6934fc522cd1a43555d88aa1baa80858c6dc99c783324c0be99edde7f1679f22
Jul 15 10:54:27.197817 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:27.197733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" event={"ID":"63aad8c8-2687-4adf-9bab-9c36b0086419","Type":"ContainerStarted","Data":"6934fc522cd1a43555d88aa1baa80858c6dc99c783324c0be99edde7f1679f22"}
Jul 15 10:54:29.763261 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:29.763212    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:54:29.763561 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:29.763298    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:54:29.763561 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:29.763335    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:54:30.215512 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:30.215464    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" event={"ID":"63aad8c8-2687-4adf-9bab-9c36b0086419","Type":"ContainerStarted","Data":"ef5f52cd1c7d1f8754c62c9e22919149e412774bc33f8e0d2a59433073ab47b3"}
Jul 15 10:54:30.236305 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:30.236261    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podStartSLOduration=1.410818809 podStartE2EDuration="4.236248118s" totalImagesPullingTime="2.825429309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:54:30.235646807 +0000 UTC m=+539.465614340" watchObservedRunningTime="2026-07-15 10:54:30.236248118 +0000 UTC m=+539.466215649"
Jul 15 10:54:30.797513 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:30.797460    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:30.802292 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:30.802265    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:31.185778 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.185699    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"]
Jul 15 10:54:31.189328 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.189307    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"
Jul 15 10:54:31.191990 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.191963    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hpbg6\""
Jul 15 10:54:31.191990 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.191983    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 15 10:54:31.192143 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.191986    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 15 10:54:31.192143 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.192037    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 15 10:54:31.192433 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.192417    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 15 10:54:31.192475 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.192424    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 15 10:54:31.201655 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.201637    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"]
Jul 15 10:54:31.219567 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.219545    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:31.220510 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.220472    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:54:31.336527 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.336479    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5-metrics-cert\") pod \"lws-controller-manager-6d9b94dcd4-b72nk\" (UID: \"d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"
Jul 15 10:54:31.336723 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.336539    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5-cert\") pod \"lws-controller-manager-6d9b94dcd4-b72nk\" (UID: \"d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"
Jul 15 10:54:31.336723 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.336652    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5-manager-config\") pod \"lws-controller-manager-6d9b94dcd4-b72nk\" (UID: \"d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"
Jul 15 10:54:31.336723 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.336670    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v4tl\" (UniqueName: \"kubernetes.io/projected/d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5-kube-api-access-7v4tl\") pod \"lws-controller-manager-6d9b94dcd4-b72nk\" (UID: \"d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"
Jul 15 10:54:31.437872 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.437774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5-manager-config\") pod \"lws-controller-manager-6d9b94dcd4-b72nk\" (UID: \"d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"
Jul 15 10:54:31.437872 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.437817    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4tl\" (UniqueName: \"kubernetes.io/projected/d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5-kube-api-access-7v4tl\") pod \"lws-controller-manager-6d9b94dcd4-b72nk\" (UID: \"d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"
Jul 15 10:54:31.438108 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.437893    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5-metrics-cert\") pod \"lws-controller-manager-6d9b94dcd4-b72nk\" (UID: \"d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"
Jul 15 10:54:31.438108 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.437935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5-cert\") pod \"lws-controller-manager-6d9b94dcd4-b72nk\" (UID: \"d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"
Jul 15 10:54:31.438415 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.438385    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5-manager-config\") pod \"lws-controller-manager-6d9b94dcd4-b72nk\" (UID: \"d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"
Jul 15 10:54:31.440409 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.440392    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5-metrics-cert\") pod \"lws-controller-manager-6d9b94dcd4-b72nk\" (UID: \"d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"
Jul 15 10:54:31.440535 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.440463    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5-cert\") pod \"lws-controller-manager-6d9b94dcd4-b72nk\" (UID: \"d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"
Jul 15 10:54:31.448225 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.448190    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v4tl\" (UniqueName: \"kubernetes.io/projected/d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5-kube-api-access-7v4tl\") pod \"lws-controller-manager-6d9b94dcd4-b72nk\" (UID: \"d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"
Jul 15 10:54:31.498298 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.498265    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"
Jul 15 10:54:31.633647 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:31.633616    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"]
Jul 15 10:54:31.636127 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:54:31.636104    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f58f48_9b6d_4577_afe4_e7ffe1f7eea5.slice/crio-91b1869ae0fba8c98d7e760f10e6d0128109cd77ee1dc67158c01becf86b4811 WatchSource:0}: Error finding container 91b1869ae0fba8c98d7e760f10e6d0128109cd77ee1dc67158c01becf86b4811: Status 404 returned error can't find the container with id 91b1869ae0fba8c98d7e760f10e6d0128109cd77ee1dc67158c01becf86b4811
Jul 15 10:54:32.225307 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:32.225271    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk" event={"ID":"d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5","Type":"ContainerStarted","Data":"91b1869ae0fba8c98d7e760f10e6d0128109cd77ee1dc67158c01becf86b4811"}
Jul 15 10:54:34.237088 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:34.237046    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk" event={"ID":"d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5","Type":"ContainerStarted","Data":"148a922f188a4cb2866e513d06990443051fe645023bdbf34c3bd097387ebb23"}
Jul 15 10:54:34.237452 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:34.237164    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"
Jul 15 10:54:34.256880 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:34.256822    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk" podStartSLOduration=0.768586976 podStartE2EDuration="3.256803649s" totalImagesPullingTime="2.488216673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:54:34.255411918 +0000 UTC m=+543.485379452" watchObservedRunningTime="2026-07-15 10:54:34.256803649 +0000 UTC m=+543.486771179"
Jul 15 10:54:45.244451 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:54:45.244422    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6d9b94dcd4-b72nk"
Jul 15 10:55:31.163775 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:55:31.163744    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 10:55:31.164287 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:55:31.163883    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 10:56:10.510373 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:10.510339    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"]
Jul 15 10:56:10.512300 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:10.512284    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"
Jul 15 10:56:10.514783 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:10.514763    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fl9gn\""
Jul 15 10:56:10.514906 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:10.514770    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 15 10:56:10.515661 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:10.515642    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 15 10:56:10.515661 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:10.515652    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 15 10:56:10.523489 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:10.523469    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"]
Jul 15 10:56:10.612587 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:10.611865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggvfd\" (UniqueName: \"kubernetes.io/projected/578f06b9-e691-48ab-aab9-ec9c9b79a2dc-kube-api-access-ggvfd\") pod \"limitador-limitador-ff68ff5cb-jk8vj\" (UID: \"578f06b9-e691-48ab-aab9-ec9c9b79a2dc\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"
Jul 15 10:56:10.612587 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:10.611981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/578f06b9-e691-48ab-aab9-ec9c9b79a2dc-config-file\") pod \"limitador-limitador-ff68ff5cb-jk8vj\" (UID: \"578f06b9-e691-48ab-aab9-ec9c9b79a2dc\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"
Jul 15 10:56:10.616519 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:10.616326    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"]
Jul 15 10:56:10.713280 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:10.713242    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggvfd\" (UniqueName: \"kubernetes.io/projected/578f06b9-e691-48ab-aab9-ec9c9b79a2dc-kube-api-access-ggvfd\") pod \"limitador-limitador-ff68ff5cb-jk8vj\" (UID: \"578f06b9-e691-48ab-aab9-ec9c9b79a2dc\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"
Jul 15 10:56:10.713447 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:10.713292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/578f06b9-e691-48ab-aab9-ec9c9b79a2dc-config-file\") pod \"limitador-limitador-ff68ff5cb-jk8vj\" (UID: \"578f06b9-e691-48ab-aab9-ec9c9b79a2dc\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"
Jul 15 10:56:10.713859 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:10.713843    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/578f06b9-e691-48ab-aab9-ec9c9b79a2dc-config-file\") pod \"limitador-limitador-ff68ff5cb-jk8vj\" (UID: \"578f06b9-e691-48ab-aab9-ec9c9b79a2dc\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"
Jul 15 10:56:10.724937 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:10.724908    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggvfd\" (UniqueName: \"kubernetes.io/projected/578f06b9-e691-48ab-aab9-ec9c9b79a2dc-kube-api-access-ggvfd\") pod \"limitador-limitador-ff68ff5cb-jk8vj\" (UID: \"578f06b9-e691-48ab-aab9-ec9c9b79a2dc\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"
Jul 15 10:56:10.822424 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:10.822337    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"
Jul 15 10:56:10.953319 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:10.953288    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"]
Jul 15 10:56:10.957973 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:56:10.957949    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578f06b9_e691_48ab_aab9_ec9c9b79a2dc.slice/crio-a8643a51b539e259c16afc460f1127e65f048b8338f4c7414aa9c2d4a4174b67 WatchSource:0}: Error finding container a8643a51b539e259c16afc460f1127e65f048b8338f4c7414aa9c2d4a4174b67: Status 404 returned error can't find the container with id a8643a51b539e259c16afc460f1127e65f048b8338f4c7414aa9c2d4a4174b67
Jul 15 10:56:11.333533 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:11.333485    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b6c89bb85-m69c4"]
Jul 15 10:56:11.336194 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:11.336177    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-m69c4"
Jul 15 10:56:11.338815 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:11.338725    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jcfqf\""
Jul 15 10:56:11.346601 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:11.346577    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-m69c4"]
Jul 15 10:56:11.418729 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:11.418696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx8mp\" (UniqueName: \"kubernetes.io/projected/467ebeb2-eb78-46b4-9390-c2c20e84177c-kube-api-access-sx8mp\") pod \"authorino-b6c89bb85-m69c4\" (UID: \"467ebeb2-eb78-46b4-9390-c2c20e84177c\") " pod="kuadrant-system/authorino-b6c89bb85-m69c4"
Jul 15 10:56:11.519609 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:11.519578    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx8mp\" (UniqueName: \"kubernetes.io/projected/467ebeb2-eb78-46b4-9390-c2c20e84177c-kube-api-access-sx8mp\") pod \"authorino-b6c89bb85-m69c4\" (UID: \"467ebeb2-eb78-46b4-9390-c2c20e84177c\") " pod="kuadrant-system/authorino-b6c89bb85-m69c4"
Jul 15 10:56:11.529323 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:11.529294    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx8mp\" (UniqueName: \"kubernetes.io/projected/467ebeb2-eb78-46b4-9390-c2c20e84177c-kube-api-access-sx8mp\") pod \"authorino-b6c89bb85-m69c4\" (UID: \"467ebeb2-eb78-46b4-9390-c2c20e84177c\") " pod="kuadrant-system/authorino-b6c89bb85-m69c4"
Jul 15 10:56:11.628170 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:11.628031    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj" event={"ID":"578f06b9-e691-48ab-aab9-ec9c9b79a2dc","Type":"ContainerStarted","Data":"a8643a51b539e259c16afc460f1127e65f048b8338f4c7414aa9c2d4a4174b67"}
Jul 15 10:56:11.645268 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:11.645138    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-m69c4"
Jul 15 10:56:11.808989 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:11.808922    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-m69c4"]
Jul 15 10:56:11.813699 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:56:11.813642    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod467ebeb2_eb78_46b4_9390_c2c20e84177c.slice/crio-0de80bd6ad076cd9469679d2befaa82d047be80776625e543454c2f94dc81a79 WatchSource:0}: Error finding container 0de80bd6ad076cd9469679d2befaa82d047be80776625e543454c2f94dc81a79: Status 404 returned error can't find the container with id 0de80bd6ad076cd9469679d2befaa82d047be80776625e543454c2f94dc81a79
Jul 15 10:56:12.635559 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:12.635526    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-m69c4" event={"ID":"467ebeb2-eb78-46b4-9390-c2c20e84177c","Type":"ContainerStarted","Data":"0de80bd6ad076cd9469679d2befaa82d047be80776625e543454c2f94dc81a79"}
Jul 15 10:56:14.646474 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:14.646438    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj" event={"ID":"578f06b9-e691-48ab-aab9-ec9c9b79a2dc","Type":"ContainerStarted","Data":"db99303d83a48d9db59c25470291ae0b75160cdcaec258ca957a440d8c18abba"}
Jul 15 10:56:14.646906 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:14.646684    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"
Jul 15 10:56:14.665277 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:14.665209    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj" podStartSLOduration=1.042123204 podStartE2EDuration="4.66518883s" totalImagesPullingTime="3.623065626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:56:14.664225326 +0000 UTC m=+643.894192856" watchObservedRunningTime="2026-07-15 10:56:14.66518883 +0000 UTC m=+643.895156362"
Jul 15 10:56:16.656988 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:16.656895    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-m69c4" event={"ID":"467ebeb2-eb78-46b4-9390-c2c20e84177c","Type":"ContainerStarted","Data":"d0cf4289f1c61cf700511f64009f942bbbfa4368da2678a673a34081a4ca23f3"}
Jul 15 10:56:16.679279 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:16.679229    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b6c89bb85-m69c4" podStartSLOduration=1.123536518 podStartE2EDuration="5.679212831s" totalImagesPullingTime="4.555676313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:56:16.679032108 +0000 UTC m=+645.908999652" watchObservedRunningTime="2026-07-15 10:56:16.679212831 +0000 UTC m=+645.909180362"
Jul 15 10:56:23.705755 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:23.705723    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-m69c4"]
Jul 15 10:56:23.706144 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:23.705890    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b6c89bb85-m69c4" podUID="467ebeb2-eb78-46b4-9390-c2c20e84177c" containerName="authorino" containerID="cri-o://d0cf4289f1c61cf700511f64009f942bbbfa4368da2678a673a34081a4ca23f3" gracePeriod=30
Jul 15 10:56:23.939634 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:23.939612    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-m69c4"
Jul 15 10:56:24.021362 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:24.021336    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx8mp\" (UniqueName: \"kubernetes.io/projected/467ebeb2-eb78-46b4-9390-c2c20e84177c-kube-api-access-sx8mp\") pod \"467ebeb2-eb78-46b4-9390-c2c20e84177c\" (UID: \"467ebeb2-eb78-46b4-9390-c2c20e84177c\") "
Jul 15 10:56:24.023688 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:24.023652    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467ebeb2-eb78-46b4-9390-c2c20e84177c-kube-api-access-sx8mp" (OuterVolumeSpecName: "kube-api-access-sx8mp") pod "467ebeb2-eb78-46b4-9390-c2c20e84177c" (UID: "467ebeb2-eb78-46b4-9390-c2c20e84177c"). InnerVolumeSpecName "kube-api-access-sx8mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:56:24.122086 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:24.122041    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx8mp\" (UniqueName: \"kubernetes.io/projected/467ebeb2-eb78-46b4-9390-c2c20e84177c-kube-api-access-sx8mp\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:56:24.690780 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:24.690750    2566 generic.go:356] "Generic (PLEG): container finished" podID="467ebeb2-eb78-46b4-9390-c2c20e84177c" containerID="d0cf4289f1c61cf700511f64009f942bbbfa4368da2678a673a34081a4ca23f3" exitCode=0
Jul 15 10:56:24.690961 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:24.690796    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-m69c4"
Jul 15 10:56:24.690961 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:24.690844    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-m69c4" event={"ID":"467ebeb2-eb78-46b4-9390-c2c20e84177c","Type":"ContainerDied","Data":"d0cf4289f1c61cf700511f64009f942bbbfa4368da2678a673a34081a4ca23f3"}
Jul 15 10:56:24.690961 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:24.690883    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-m69c4" event={"ID":"467ebeb2-eb78-46b4-9390-c2c20e84177c","Type":"ContainerDied","Data":"0de80bd6ad076cd9469679d2befaa82d047be80776625e543454c2f94dc81a79"}
Jul 15 10:56:24.690961 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:24.690913    2566 scope.go:117] "RemoveContainer" containerID="d0cf4289f1c61cf700511f64009f942bbbfa4368da2678a673a34081a4ca23f3"
Jul 15 10:56:24.699521 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:24.699486    2566 scope.go:117] "RemoveContainer" containerID="d0cf4289f1c61cf700511f64009f942bbbfa4368da2678a673a34081a4ca23f3"
Jul 15 10:56:24.699785 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:56:24.699767    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0cf4289f1c61cf700511f64009f942bbbfa4368da2678a673a34081a4ca23f3\": container with ID starting with d0cf4289f1c61cf700511f64009f942bbbfa4368da2678a673a34081a4ca23f3 not found: ID does not exist" containerID="d0cf4289f1c61cf700511f64009f942bbbfa4368da2678a673a34081a4ca23f3"
Jul 15 10:56:24.699831 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:24.699795    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0cf4289f1c61cf700511f64009f942bbbfa4368da2678a673a34081a4ca23f3"} err="failed to get container status \"d0cf4289f1c61cf700511f64009f942bbbfa4368da2678a673a34081a4ca23f3\": rpc error: code = NotFound desc = could not find container \"d0cf4289f1c61cf700511f64009f942bbbfa4368da2678a673a34081a4ca23f3\": container with ID starting with d0cf4289f1c61cf700511f64009f942bbbfa4368da2678a673a34081a4ca23f3 not found: ID does not exist"
Jul 15 10:56:24.719752 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:24.719721    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-m69c4"]
Jul 15 10:56:24.722807 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:24.722781    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-m69c4"]
Jul 15 10:56:25.268318 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:25.268284    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="467ebeb2-eb78-46b4-9390-c2c20e84177c" path="/var/lib/kubelet/pods/467ebeb2-eb78-46b4-9390-c2c20e84177c/volumes"
Jul 15 10:56:25.652768 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:25.652693    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"
Jul 15 10:56:26.304322 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:26.304295    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"]
Jul 15 10:56:26.304822 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:26.304460    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj" podUID="578f06b9-e691-48ab-aab9-ec9c9b79a2dc" containerName="limitador" containerID="cri-o://db99303d83a48d9db59c25470291ae0b75160cdcaec258ca957a440d8c18abba" gracePeriod=30
Jul 15 10:56:26.851916 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:26.851893    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"
Jul 15 10:56:26.946360 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:26.946278    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggvfd\" (UniqueName: \"kubernetes.io/projected/578f06b9-e691-48ab-aab9-ec9c9b79a2dc-kube-api-access-ggvfd\") pod \"578f06b9-e691-48ab-aab9-ec9c9b79a2dc\" (UID: \"578f06b9-e691-48ab-aab9-ec9c9b79a2dc\") "
Jul 15 10:56:26.946360 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:26.946357    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/578f06b9-e691-48ab-aab9-ec9c9b79a2dc-config-file\") pod \"578f06b9-e691-48ab-aab9-ec9c9b79a2dc\" (UID: \"578f06b9-e691-48ab-aab9-ec9c9b79a2dc\") "
Jul 15 10:56:26.946737 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:26.946713    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/578f06b9-e691-48ab-aab9-ec9c9b79a2dc-config-file" (OuterVolumeSpecName: "config-file") pod "578f06b9-e691-48ab-aab9-ec9c9b79a2dc" (UID: "578f06b9-e691-48ab-aab9-ec9c9b79a2dc"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:56:26.948484 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:26.948457    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578f06b9-e691-48ab-aab9-ec9c9b79a2dc-kube-api-access-ggvfd" (OuterVolumeSpecName: "kube-api-access-ggvfd") pod "578f06b9-e691-48ab-aab9-ec9c9b79a2dc" (UID: "578f06b9-e691-48ab-aab9-ec9c9b79a2dc"). InnerVolumeSpecName "kube-api-access-ggvfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:56:27.047239 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:27.047203    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggvfd\" (UniqueName: \"kubernetes.io/projected/578f06b9-e691-48ab-aab9-ec9c9b79a2dc-kube-api-access-ggvfd\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:56:27.047239 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:27.047232    2566 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/578f06b9-e691-48ab-aab9-ec9c9b79a2dc-config-file\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:56:27.705449 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:27.705363    2566 generic.go:356] "Generic (PLEG): container finished" podID="578f06b9-e691-48ab-aab9-ec9c9b79a2dc" containerID="db99303d83a48d9db59c25470291ae0b75160cdcaec258ca957a440d8c18abba" exitCode=0
Jul 15 10:56:27.705449 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:27.705409    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"
Jul 15 10:56:27.705449 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:27.705435    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj" event={"ID":"578f06b9-e691-48ab-aab9-ec9c9b79a2dc","Type":"ContainerDied","Data":"db99303d83a48d9db59c25470291ae0b75160cdcaec258ca957a440d8c18abba"}
Jul 15 10:56:27.705970 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:27.705478    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj" event={"ID":"578f06b9-e691-48ab-aab9-ec9c9b79a2dc","Type":"ContainerDied","Data":"a8643a51b539e259c16afc460f1127e65f048b8338f4c7414aa9c2d4a4174b67"}
Jul 15 10:56:27.705970 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:27.705526    2566 scope.go:117] "RemoveContainer" containerID="db99303d83a48d9db59c25470291ae0b75160cdcaec258ca957a440d8c18abba"
Jul 15 10:56:27.713482 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:27.713464    2566 scope.go:117] "RemoveContainer" containerID="db99303d83a48d9db59c25470291ae0b75160cdcaec258ca957a440d8c18abba"
Jul 15 10:56:27.713787 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:56:27.713766    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db99303d83a48d9db59c25470291ae0b75160cdcaec258ca957a440d8c18abba\": container with ID starting with db99303d83a48d9db59c25470291ae0b75160cdcaec258ca957a440d8c18abba not found: ID does not exist" containerID="db99303d83a48d9db59c25470291ae0b75160cdcaec258ca957a440d8c18abba"
Jul 15 10:56:27.713862 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:27.713800    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db99303d83a48d9db59c25470291ae0b75160cdcaec258ca957a440d8c18abba"} err="failed to get container status \"db99303d83a48d9db59c25470291ae0b75160cdcaec258ca957a440d8c18abba\": rpc error: code = NotFound desc = could not find container \"db99303d83a48d9db59c25470291ae0b75160cdcaec258ca957a440d8c18abba\": container with ID starting with db99303d83a48d9db59c25470291ae0b75160cdcaec258ca957a440d8c18abba not found: ID does not exist"
Jul 15 10:56:27.724110 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:27.724090    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"]
Jul 15 10:56:27.728889 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:27.728871    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-jk8vj"]
Jul 15 10:56:29.267232 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:29.267202    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="578f06b9-e691-48ab-aab9-ec9c9b79a2dc" path="/var/lib/kubelet/pods/578f06b9-e691-48ab-aab9-ec9c9b79a2dc/volumes"
Jul 15 10:56:36.464395 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.464355    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c77957575-twxjh"]
Jul 15 10:56:36.464899 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.464797    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="578f06b9-e691-48ab-aab9-ec9c9b79a2dc" containerName="limitador"
Jul 15 10:56:36.464899 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.464815    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="578f06b9-e691-48ab-aab9-ec9c9b79a2dc" containerName="limitador"
Jul 15 10:56:36.464899 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.464827    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="467ebeb2-eb78-46b4-9390-c2c20e84177c" containerName="authorino"
Jul 15 10:56:36.464899 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.464835    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="467ebeb2-eb78-46b4-9390-c2c20e84177c" containerName="authorino"
Jul 15 10:56:36.465111 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.464934    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="578f06b9-e691-48ab-aab9-ec9c9b79a2dc" containerName="limitador"
Jul 15 10:56:36.465111 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.464945    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="467ebeb2-eb78-46b4-9390-c2c20e84177c" containerName="authorino"
Jul 15 10:56:36.471804 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.471782    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-twxjh"
Jul 15 10:56:36.475700 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.475667    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jcfqf\""
Jul 15 10:56:36.475700 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.475690    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 15 10:56:36.475866 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.475747    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 15 10:56:36.475866 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.475690    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 15 10:56:36.485285 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.485261    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-twxjh"]
Jul 15 10:56:36.627251 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.627212    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aab5fd40-59b4-4d7f-97d9-4ccf3a6f0c6d-tls-cert\") pod \"authorino-7c77957575-twxjh\" (UID: \"aab5fd40-59b4-4d7f-97d9-4ccf3a6f0c6d\") " pod="kuadrant-system/authorino-7c77957575-twxjh"
Jul 15 10:56:36.627417 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.627259    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlfjz\" (UniqueName: \"kubernetes.io/projected/aab5fd40-59b4-4d7f-97d9-4ccf3a6f0c6d-kube-api-access-zlfjz\") pod \"authorino-7c77957575-twxjh\" (UID: \"aab5fd40-59b4-4d7f-97d9-4ccf3a6f0c6d\") " pod="kuadrant-system/authorino-7c77957575-twxjh"
Jul 15 10:56:36.728768 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.728682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aab5fd40-59b4-4d7f-97d9-4ccf3a6f0c6d-tls-cert\") pod \"authorino-7c77957575-twxjh\" (UID: \"aab5fd40-59b4-4d7f-97d9-4ccf3a6f0c6d\") " pod="kuadrant-system/authorino-7c77957575-twxjh"
Jul 15 10:56:36.728768 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.728737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlfjz\" (UniqueName: \"kubernetes.io/projected/aab5fd40-59b4-4d7f-97d9-4ccf3a6f0c6d-kube-api-access-zlfjz\") pod \"authorino-7c77957575-twxjh\" (UID: \"aab5fd40-59b4-4d7f-97d9-4ccf3a6f0c6d\") " pod="kuadrant-system/authorino-7c77957575-twxjh"
Jul 15 10:56:36.731228 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.731206    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aab5fd40-59b4-4d7f-97d9-4ccf3a6f0c6d-tls-cert\") pod \"authorino-7c77957575-twxjh\" (UID: \"aab5fd40-59b4-4d7f-97d9-4ccf3a6f0c6d\") " pod="kuadrant-system/authorino-7c77957575-twxjh"
Jul 15 10:56:36.738638 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.738615    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlfjz\" (UniqueName: \"kubernetes.io/projected/aab5fd40-59b4-4d7f-97d9-4ccf3a6f0c6d-kube-api-access-zlfjz\") pod \"authorino-7c77957575-twxjh\" (UID: \"aab5fd40-59b4-4d7f-97d9-4ccf3a6f0c6d\") " pod="kuadrant-system/authorino-7c77957575-twxjh"
Jul 15 10:56:36.780653 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.780625    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-twxjh"
Jul 15 10:56:36.919464 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:36.919426    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-twxjh"]
Jul 15 10:56:36.922751 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:56:36.922725    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaab5fd40_59b4_4d7f_97d9_4ccf3a6f0c6d.slice/crio-0225235f3d35cc2cae92dadd3deede85a408b939b837161a56d56dc8a192e707 WatchSource:0}: Error finding container 0225235f3d35cc2cae92dadd3deede85a408b939b837161a56d56dc8a192e707: Status 404 returned error can't find the container with id 0225235f3d35cc2cae92dadd3deede85a408b939b837161a56d56dc8a192e707
Jul 15 10:56:37.749084 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:37.749039    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-twxjh" event={"ID":"aab5fd40-59b4-4d7f-97d9-4ccf3a6f0c6d","Type":"ContainerStarted","Data":"00fab5e422b09576c9eb6c714601c3d8c78a1a64cbe76aae1919cff75d1fd998"}
Jul 15 10:56:37.749428 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:37.749098    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-twxjh" event={"ID":"aab5fd40-59b4-4d7f-97d9-4ccf3a6f0c6d","Type":"ContainerStarted","Data":"0225235f3d35cc2cae92dadd3deede85a408b939b837161a56d56dc8a192e707"}
Jul 15 10:56:37.770014 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:37.769956    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c77957575-twxjh" podStartSLOduration=1.067363026 podStartE2EDuration="1.769939981s" totalImagesPullingTime="702.576955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:56:37.768044027 +0000 UTC m=+666.998011556" watchObservedRunningTime="2026-07-15 10:56:37.769939981 +0000 UTC m=+666.999907512"
Jul 15 10:56:42.354123 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.354093    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-749567458d-svt6l"]
Jul 15 10:56:42.354674 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.354274    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-749567458d-svt6l" podUID="3285dfeb-997a-409b-afac-43462bdb37f1" containerName="manager" containerID="cri-o://935b133a0b073803d3fc97fb48e0e2f06fd0a2123e23bbc7671a1d6e2baab87f" gracePeriod=10
Jul 15 10:56:42.379219 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.379196    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-749567458d-rwq52"]
Jul 15 10:56:42.382103 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.382087    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-749567458d-rwq52"
Jul 15 10:56:42.393955 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.393933    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-749567458d-rwq52"]
Jul 15 10:56:42.475194 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.475155    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdcdr\" (UniqueName: \"kubernetes.io/projected/3049f489-4d0e-45cd-8cda-3adae22e5975-kube-api-access-pdcdr\") pod \"kserve-controller-manager-749567458d-rwq52\" (UID: \"3049f489-4d0e-45cd-8cda-3adae22e5975\") " pod="kserve/kserve-controller-manager-749567458d-rwq52"
Jul 15 10:56:42.475312 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.475210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3049f489-4d0e-45cd-8cda-3adae22e5975-cert\") pod \"kserve-controller-manager-749567458d-rwq52\" (UID: \"3049f489-4d0e-45cd-8cda-3adae22e5975\") " pod="kserve/kserve-controller-manager-749567458d-rwq52"
Jul 15 10:56:42.576426 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.576391    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdcdr\" (UniqueName: \"kubernetes.io/projected/3049f489-4d0e-45cd-8cda-3adae22e5975-kube-api-access-pdcdr\") pod \"kserve-controller-manager-749567458d-rwq52\" (UID: \"3049f489-4d0e-45cd-8cda-3adae22e5975\") " pod="kserve/kserve-controller-manager-749567458d-rwq52"
Jul 15 10:56:42.576625 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.576445    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3049f489-4d0e-45cd-8cda-3adae22e5975-cert\") pod \"kserve-controller-manager-749567458d-rwq52\" (UID: \"3049f489-4d0e-45cd-8cda-3adae22e5975\") " pod="kserve/kserve-controller-manager-749567458d-rwq52"
Jul 15 10:56:42.600333 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.600151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3049f489-4d0e-45cd-8cda-3adae22e5975-cert\") pod \"kserve-controller-manager-749567458d-rwq52\" (UID: \"3049f489-4d0e-45cd-8cda-3adae22e5975\") " pod="kserve/kserve-controller-manager-749567458d-rwq52"
Jul 15 10:56:42.602814 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.600546    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdcdr\" (UniqueName: \"kubernetes.io/projected/3049f489-4d0e-45cd-8cda-3adae22e5975-kube-api-access-pdcdr\") pod \"kserve-controller-manager-749567458d-rwq52\" (UID: \"3049f489-4d0e-45cd-8cda-3adae22e5975\") " pod="kserve/kserve-controller-manager-749567458d-rwq52"
Jul 15 10:56:42.623791 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.623765    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-749567458d-svt6l"
Jul 15 10:56:42.730778 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.730746    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-749567458d-rwq52"
Jul 15 10:56:42.770073 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.770041    2566 generic.go:356] "Generic (PLEG): container finished" podID="3285dfeb-997a-409b-afac-43462bdb37f1" containerID="935b133a0b073803d3fc97fb48e0e2f06fd0a2123e23bbc7671a1d6e2baab87f" exitCode=0
Jul 15 10:56:42.770253 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.770079    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-749567458d-svt6l"
Jul 15 10:56:42.770253 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.770131    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-749567458d-svt6l" event={"ID":"3285dfeb-997a-409b-afac-43462bdb37f1","Type":"ContainerDied","Data":"935b133a0b073803d3fc97fb48e0e2f06fd0a2123e23bbc7671a1d6e2baab87f"}
Jul 15 10:56:42.770253 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.770190    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-749567458d-svt6l" event={"ID":"3285dfeb-997a-409b-afac-43462bdb37f1","Type":"ContainerDied","Data":"7e23aaaf5bb2afe2893fd06edab3551ff17932dd5f881453c14da6dda70a1178"}
Jul 15 10:56:42.770253 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.770224    2566 scope.go:117] "RemoveContainer" containerID="935b133a0b073803d3fc97fb48e0e2f06fd0a2123e23bbc7671a1d6e2baab87f"
Jul 15 10:56:42.778289 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.778270    2566 scope.go:117] "RemoveContainer" containerID="935b133a0b073803d3fc97fb48e0e2f06fd0a2123e23bbc7671a1d6e2baab87f"
Jul 15 10:56:42.778359 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.778300    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3285dfeb-997a-409b-afac-43462bdb37f1-cert\") pod \"3285dfeb-997a-409b-afac-43462bdb37f1\" (UID: \"3285dfeb-997a-409b-afac-43462bdb37f1\") "
Jul 15 10:56:42.778426 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.778407    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29vhj\" (UniqueName: \"kubernetes.io/projected/3285dfeb-997a-409b-afac-43462bdb37f1-kube-api-access-29vhj\") pod \"3285dfeb-997a-409b-afac-43462bdb37f1\" (UID: \"3285dfeb-997a-409b-afac-43462bdb37f1\") "
Jul 15 10:56:42.778649 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:56:42.778626    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935b133a0b073803d3fc97fb48e0e2f06fd0a2123e23bbc7671a1d6e2baab87f\": container with ID starting with 935b133a0b073803d3fc97fb48e0e2f06fd0a2123e23bbc7671a1d6e2baab87f not found: ID does not exist" containerID="935b133a0b073803d3fc97fb48e0e2f06fd0a2123e23bbc7671a1d6e2baab87f"
Jul 15 10:56:42.778743 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.778672    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935b133a0b073803d3fc97fb48e0e2f06fd0a2123e23bbc7671a1d6e2baab87f"} err="failed to get container status \"935b133a0b073803d3fc97fb48e0e2f06fd0a2123e23bbc7671a1d6e2baab87f\": rpc error: code = NotFound desc = could not find container \"935b133a0b073803d3fc97fb48e0e2f06fd0a2123e23bbc7671a1d6e2baab87f\": container with ID starting with 935b133a0b073803d3fc97fb48e0e2f06fd0a2123e23bbc7671a1d6e2baab87f not found: ID does not exist"
Jul 15 10:56:42.781132 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.781106    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3285dfeb-997a-409b-afac-43462bdb37f1-cert" (OuterVolumeSpecName: "cert") pod "3285dfeb-997a-409b-afac-43462bdb37f1" (UID: "3285dfeb-997a-409b-afac-43462bdb37f1"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:56:42.781395 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.781342    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3285dfeb-997a-409b-afac-43462bdb37f1-kube-api-access-29vhj" (OuterVolumeSpecName: "kube-api-access-29vhj") pod "3285dfeb-997a-409b-afac-43462bdb37f1" (UID: "3285dfeb-997a-409b-afac-43462bdb37f1"). InnerVolumeSpecName "kube-api-access-29vhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:56:42.862932 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.862908    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-749567458d-rwq52"]
Jul 15 10:56:42.865151 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:56:42.865123    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3049f489_4d0e_45cd_8cda_3adae22e5975.slice/crio-ff88cc8f993958c1c47e37384cec93387cd209e81213a5b7812e2cbd6886142b WatchSource:0}: Error finding container ff88cc8f993958c1c47e37384cec93387cd209e81213a5b7812e2cbd6886142b: Status 404 returned error can't find the container with id ff88cc8f993958c1c47e37384cec93387cd209e81213a5b7812e2cbd6886142b
Jul 15 10:56:42.879610 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.879591    2566 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3285dfeb-997a-409b-afac-43462bdb37f1-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:56:42.879706 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:42.879616    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29vhj\" (UniqueName: \"kubernetes.io/projected/3285dfeb-997a-409b-afac-43462bdb37f1-kube-api-access-29vhj\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:56:43.091165 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:43.091129    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-749567458d-svt6l"]
Jul 15 10:56:43.098331 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:43.098302    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-749567458d-svt6l"]
Jul 15 10:56:43.268304 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:43.268277    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3285dfeb-997a-409b-afac-43462bdb37f1" path="/var/lib/kubelet/pods/3285dfeb-997a-409b-afac-43462bdb37f1/volumes"
Jul 15 10:56:43.775187 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:43.775155    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-749567458d-rwq52" event={"ID":"3049f489-4d0e-45cd-8cda-3adae22e5975","Type":"ContainerStarted","Data":"aed288e084a9c06a12c3a969263466743f9f06a6d457038006a706eee9dbc3a7"}
Jul 15 10:56:43.775629 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:43.775203    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-749567458d-rwq52" event={"ID":"3049f489-4d0e-45cd-8cda-3adae22e5975","Type":"ContainerStarted","Data":"ff88cc8f993958c1c47e37384cec93387cd209e81213a5b7812e2cbd6886142b"}
Jul 15 10:56:43.775629 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:43.775281    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-749567458d-rwq52"
Jul 15 10:56:43.796920 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:56:43.796858    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-749567458d-rwq52" podStartSLOduration=1.414776617 podStartE2EDuration="1.796840622s" totalImagesPullingTime="382.064005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:56:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:56:43.793656829 +0000 UTC m=+673.023624361" watchObservedRunningTime="2026-07-15 10:56:43.796840622 +0000 UTC m=+673.026808155"
Jul 15 10:57:14.787043 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:14.787015    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-749567458d-rwq52"
Jul 15 10:57:31.912505 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:31.912462    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-9thnn"]
Jul 15 10:57:31.912861 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:31.912772    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3285dfeb-997a-409b-afac-43462bdb37f1" containerName="manager"
Jul 15 10:57:31.912861 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:31.912783    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3285dfeb-997a-409b-afac-43462bdb37f1" containerName="manager"
Jul 15 10:57:31.912861 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:31.912848    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3285dfeb-997a-409b-afac-43462bdb37f1" containerName="manager"
Jul 15 10:57:31.921268 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:31.921240    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9thnn"
Jul 15 10:57:31.923210 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:31.923182    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9thnn"]
Jul 15 10:57:32.067927 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:32.067885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7mgq\" (UniqueName: \"kubernetes.io/projected/a476ce1b-19f5-42df-a54f-cd25aa1dbb66-kube-api-access-c7mgq\") pod \"s3-init-9thnn\" (UID: \"a476ce1b-19f5-42df-a54f-cd25aa1dbb66\") " pod="kserve/s3-init-9thnn"
Jul 15 10:57:32.068099 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:32.067939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a476ce1b-19f5-42df-a54f-cd25aa1dbb66-model-cache\") pod \"s3-init-9thnn\" (UID: \"a476ce1b-19f5-42df-a54f-cd25aa1dbb66\") " pod="kserve/s3-init-9thnn"
Jul 15 10:57:32.169326 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:32.169250    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a476ce1b-19f5-42df-a54f-cd25aa1dbb66-model-cache\") pod \"s3-init-9thnn\" (UID: \"a476ce1b-19f5-42df-a54f-cd25aa1dbb66\") " pod="kserve/s3-init-9thnn"
Jul 15 10:57:32.169326 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:32.169326    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7mgq\" (UniqueName: \"kubernetes.io/projected/a476ce1b-19f5-42df-a54f-cd25aa1dbb66-kube-api-access-c7mgq\") pod \"s3-init-9thnn\" (UID: \"a476ce1b-19f5-42df-a54f-cd25aa1dbb66\") " pod="kserve/s3-init-9thnn"
Jul 15 10:57:32.169666 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:32.169646    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a476ce1b-19f5-42df-a54f-cd25aa1dbb66-model-cache\") pod \"s3-init-9thnn\" (UID: \"a476ce1b-19f5-42df-a54f-cd25aa1dbb66\") " pod="kserve/s3-init-9thnn"
Jul 15 10:57:32.179639 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:32.179617    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7mgq\" (UniqueName: \"kubernetes.io/projected/a476ce1b-19f5-42df-a54f-cd25aa1dbb66-kube-api-access-c7mgq\") pod \"s3-init-9thnn\" (UID: \"a476ce1b-19f5-42df-a54f-cd25aa1dbb66\") " pod="kserve/s3-init-9thnn"
Jul 15 10:57:32.232460 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:32.232430    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9thnn"
Jul 15 10:57:32.362374 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:32.362340    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9thnn"]
Jul 15 10:57:32.364978 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:57:32.364952    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda476ce1b_19f5_42df_a54f_cd25aa1dbb66.slice/crio-c96b6ffe08186448657c23ca507b225393308e4c1cc8b90db577023da59121c0 WatchSource:0}: Error finding container c96b6ffe08186448657c23ca507b225393308e4c1cc8b90db577023da59121c0: Status 404 returned error can't find the container with id c96b6ffe08186448657c23ca507b225393308e4c1cc8b90db577023da59121c0
Jul 15 10:57:32.980561 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:32.980524    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9thnn" event={"ID":"a476ce1b-19f5-42df-a54f-cd25aa1dbb66","Type":"ContainerStarted","Data":"c96b6ffe08186448657c23ca507b225393308e4c1cc8b90db577023da59121c0"}
Jul 15 10:57:35.995644 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:35.995606    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9thnn" event={"ID":"a476ce1b-19f5-42df-a54f-cd25aa1dbb66","Type":"ContainerStarted","Data":"ce41af6b9cdc517da6ae2103f6471448882e89abf93f3a7677fd7bed93140d5d"}
Jul 15 10:57:37.001037 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:37.000998    2566 generic.go:356] "Generic (PLEG): container finished" podID="a476ce1b-19f5-42df-a54f-cd25aa1dbb66" containerID="ce41af6b9cdc517da6ae2103f6471448882e89abf93f3a7677fd7bed93140d5d" exitCode=0
Jul 15 10:57:37.001416 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:37.001052    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9thnn" event={"ID":"a476ce1b-19f5-42df-a54f-cd25aa1dbb66","Type":"ContainerDied","Data":"ce41af6b9cdc517da6ae2103f6471448882e89abf93f3a7677fd7bed93140d5d"}
Jul 15 10:57:42.028441 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:42.028394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9thnn" event={"ID":"a476ce1b-19f5-42df-a54f-cd25aa1dbb66","Type":"ContainerStarted","Data":"4b80932ae442ecb54a05880b96b18056a9283392e783d16eae8a89322206006e"}
Jul 15 10:57:42.047327 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:57:42.047270    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-9thnn" podStartSLOduration=2.27885426 podStartE2EDuration="11.047250239s" totalImagesPullingTime="7.751810674s" totalInitContainerRuntime="1.016585305s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:57:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:57:42.045605654 +0000 UTC m=+731.275573184" watchObservedRunningTime="2026-07-15 10:57:42.047250239 +0000 UTC m=+731.277217770"
Jul 15 10:58:47.295401 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:58:47.295365    2566 generic.go:356] "Generic (PLEG): container finished" podID="a476ce1b-19f5-42df-a54f-cd25aa1dbb66" containerID="4b80932ae442ecb54a05880b96b18056a9283392e783d16eae8a89322206006e" exitCode=0
Jul 15 10:58:47.295836 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:58:47.295438    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9thnn" event={"ID":"a476ce1b-19f5-42df-a54f-cd25aa1dbb66","Type":"ContainerDied","Data":"4b80932ae442ecb54a05880b96b18056a9283392e783d16eae8a89322206006e"}
Jul 15 10:58:48.425694 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:58:48.425669    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9thnn"
Jul 15 10:58:48.587539 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:58:48.587423    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7mgq\" (UniqueName: \"kubernetes.io/projected/a476ce1b-19f5-42df-a54f-cd25aa1dbb66-kube-api-access-c7mgq\") pod \"a476ce1b-19f5-42df-a54f-cd25aa1dbb66\" (UID: \"a476ce1b-19f5-42df-a54f-cd25aa1dbb66\") "
Jul 15 10:58:48.587539 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:58:48.587467    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a476ce1b-19f5-42df-a54f-cd25aa1dbb66-model-cache\") pod \"a476ce1b-19f5-42df-a54f-cd25aa1dbb66\" (UID: \"a476ce1b-19f5-42df-a54f-cd25aa1dbb66\") "
Jul 15 10:58:48.589769 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:58:48.589729    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a476ce1b-19f5-42df-a54f-cd25aa1dbb66-kube-api-access-c7mgq" (OuterVolumeSpecName: "kube-api-access-c7mgq") pod "a476ce1b-19f5-42df-a54f-cd25aa1dbb66" (UID: "a476ce1b-19f5-42df-a54f-cd25aa1dbb66"). InnerVolumeSpecName "kube-api-access-c7mgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:58:48.606388 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:58:48.606358    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a476ce1b-19f5-42df-a54f-cd25aa1dbb66-model-cache" (OuterVolumeSpecName: "model-cache") pod "a476ce1b-19f5-42df-a54f-cd25aa1dbb66" (UID: "a476ce1b-19f5-42df-a54f-cd25aa1dbb66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:58:48.688169 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:58:48.688138    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a476ce1b-19f5-42df-a54f-cd25aa1dbb66-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:58:48.688169 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:58:48.688166    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7mgq\" (UniqueName: \"kubernetes.io/projected/a476ce1b-19f5-42df-a54f-cd25aa1dbb66-kube-api-access-c7mgq\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:58:49.304593 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:58:49.304568    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9thnn"
Jul 15 10:58:49.304738 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:58:49.304562    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9thnn" event={"ID":"a476ce1b-19f5-42df-a54f-cd25aa1dbb66","Type":"ContainerDied","Data":"c96b6ffe08186448657c23ca507b225393308e4c1cc8b90db577023da59121c0"}
Jul 15 10:58:49.304738 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:58:49.304680    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c96b6ffe08186448657c23ca507b225393308e4c1cc8b90db577023da59121c0"
Jul 15 10:59:03.036053 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.036017    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"]
Jul 15 10:59:03.036486 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.036467    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a476ce1b-19f5-42df-a54f-cd25aa1dbb66" containerName="download-hf-model"
Jul 15 10:59:03.036592 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.036489    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a476ce1b-19f5-42df-a54f-cd25aa1dbb66" containerName="download-hf-model"
Jul 15 10:59:03.036592 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.036541    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a476ce1b-19f5-42df-a54f-cd25aa1dbb66" containerName="s3-init"
Jul 15 10:59:03.036592 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.036550    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a476ce1b-19f5-42df-a54f-cd25aa1dbb66" containerName="s3-init"
Jul 15 10:59:03.036738 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.036629    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a476ce1b-19f5-42df-a54f-cd25aa1dbb66" containerName="s3-init"
Jul 15 10:59:03.039593 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.039574    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.048980 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.048961    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 15 10:59:03.049083 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.049030    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-95m52\""
Jul 15 10:59:03.049838 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.049822    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 15 10:59:03.049932 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.049859    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 15 10:59:03.066565 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.066540    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"]
Jul 15 10:59:03.107367 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.107331    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.107569 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.107377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.107569 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.107401    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.107569 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.107438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.107569 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.107455    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.107569 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.107533    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.107569 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.107557    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.107759 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.107591    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.107759 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.107614    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlzlq\" (UniqueName: \"kubernetes.io/projected/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-kube-api-access-hlzlq\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.207994 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.207942    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.207994 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.207992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.208263 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.208021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.208263 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.208042    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.208263 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.208071    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.208263 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.208095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlzlq\" (UniqueName: \"kubernetes.io/projected/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-kube-api-access-hlzlq\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.208464 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.208265    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.208544 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.208474    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.208598 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.208588    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.208651 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.208596    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.208651 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.208620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.208916 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.208855    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.208916 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.208879    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.209098 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.209074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.210478 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.210454    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.210673 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.210656    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.216626 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.216602    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.216732 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.216677    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlzlq\" (UniqueName: \"kubernetes.io/projected/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-kube-api-access-hlzlq\") pod \"router-gateway-1-openshift-default-6679d9568-p8gmc\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.349431 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.349345    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:03.485385 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.485358    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"]
Jul 15 10:59:03.487909 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:59:03.487881    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e602c3c_f7fd_433c_b667_ec0f0ba3278c.slice/crio-25d605163f340202f850641eb4d3079f37f539d8949645de81e9744a56f494fa WatchSource:0}: Error finding container 25d605163f340202f850641eb4d3079f37f539d8949645de81e9744a56f494fa: Status 404 returned error can't find the container with id 25d605163f340202f850641eb4d3079f37f539d8949645de81e9744a56f494fa
Jul 15 10:59:03.489797 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.489763    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:59:03.489889 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.489832    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:59:03.489889 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:03.489865    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:59:04.373989 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:04.373950    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc" event={"ID":"8e602c3c-f7fd-433c-b667-ec0f0ba3278c","Type":"ContainerStarted","Data":"9dfac37137eb6ecd6da362fb6312a27ee1a0b3a94000ea2be62b621448832cde"}
Jul 15 10:59:04.374373 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:04.374013    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc" event={"ID":"8e602c3c-f7fd-433c-b667-ec0f0ba3278c","Type":"ContainerStarted","Data":"25d605163f340202f850641eb4d3079f37f539d8949645de81e9744a56f494fa"}
Jul 15 10:59:04.397251 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:04.397206    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc" podStartSLOduration=1.3971867310000001 podStartE2EDuration="1.397186731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:59:04.396267322 +0000 UTC m=+813.626234853" watchObservedRunningTime="2026-07-15 10:59:04.397186731 +0000 UTC m=+813.627154262"
Jul 15 10:59:05.349783 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:05.349749    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:05.354697 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:05.354675    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:05.378758 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:05.378738    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:05.379990 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:05.379963    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:08.600991 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:08.600961    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"]
Jul 15 10:59:08.601374 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:08.601156    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc" podUID="8e602c3c-f7fd-433c-b667-ec0f0ba3278c" containerName="istio-proxy" containerID="cri-o://9dfac37137eb6ecd6da362fb6312a27ee1a0b3a94000ea2be62b621448832cde" gracePeriod=30
Jul 15 10:59:13.848285 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.848259    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:13.899541 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.899489    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-token\") pod \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") "
Jul 15 10:59:13.899682 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.899571    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-workload-socket\") pod \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") "
Jul 15 10:59:13.899682 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.899607    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-envoy\") pod \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") "
Jul 15 10:59:13.899682 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.899640    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-workload-certs\") pod \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") "
Jul 15 10:59:13.899682 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.899666    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-podinfo\") pod \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") "
Jul 15 10:59:13.899883 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.899693    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istiod-ca-cert\") pod \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") "
Jul 15 10:59:13.899883 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.899746    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-data\") pod \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") "
Jul 15 10:59:13.899883 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.899795    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-credential-socket\") pod \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") "
Jul 15 10:59:13.899883 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.899821    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlzlq\" (UniqueName: \"kubernetes.io/projected/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-kube-api-access-hlzlq\") pod \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\" (UID: \"8e602c3c-f7fd-433c-b667-ec0f0ba3278c\") "
Jul 15 10:59:13.900063 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.899909    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e602c3c-f7fd-433c-b667-ec0f0ba3278c" (UID: "8e602c3c-f7fd-433c-b667-ec0f0ba3278c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:13.900063 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.899930    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e602c3c-f7fd-433c-b667-ec0f0ba3278c" (UID: "8e602c3c-f7fd-433c-b667-ec0f0ba3278c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:13.900063 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.900040    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:13.900063 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.900058    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:13.900215 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.900072    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e602c3c-f7fd-433c-b667-ec0f0ba3278c" (UID: "8e602c3c-f7fd-433c-b667-ec0f0ba3278c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:13.900215 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.900164    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e602c3c-f7fd-433c-b667-ec0f0ba3278c" (UID: "8e602c3c-f7fd-433c-b667-ec0f0ba3278c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:13.900754 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.900691    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e602c3c-f7fd-433c-b667-ec0f0ba3278c" (UID: "8e602c3c-f7fd-433c-b667-ec0f0ba3278c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:59:13.901909 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.901886    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e602c3c-f7fd-433c-b667-ec0f0ba3278c" (UID: "8e602c3c-f7fd-433c-b667-ec0f0ba3278c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:59:13.901998 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.901935    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e602c3c-f7fd-433c-b667-ec0f0ba3278c" (UID: "8e602c3c-f7fd-433c-b667-ec0f0ba3278c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:13.901998 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.901940    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e602c3c-f7fd-433c-b667-ec0f0ba3278c" (UID: "8e602c3c-f7fd-433c-b667-ec0f0ba3278c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 10:59:13.902220 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:13.902205    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-kube-api-access-hlzlq" (OuterVolumeSpecName: "kube-api-access-hlzlq") pod "8e602c3c-f7fd-433c-b667-ec0f0ba3278c" (UID: "8e602c3c-f7fd-433c-b667-ec0f0ba3278c"). InnerVolumeSpecName "kube-api-access-hlzlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:59:14.000727 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:14.000696    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:14.000727 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:14.000723    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlzlq\" (UniqueName: \"kubernetes.io/projected/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-kube-api-access-hlzlq\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:14.000901 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:14.000738    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:14.000901 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:14.000751    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:14.000901 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:14.000761    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:14.000901 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:14.000771    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:14.000901 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:14.000781    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e602c3c-f7fd-433c-b667-ec0f0ba3278c-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:14.415815 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:14.415781    2566 generic.go:356] "Generic (PLEG): container finished" podID="8e602c3c-f7fd-433c-b667-ec0f0ba3278c" containerID="9dfac37137eb6ecd6da362fb6312a27ee1a0b3a94000ea2be62b621448832cde" exitCode=0
Jul 15 10:59:14.415815 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:14.415820    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc" event={"ID":"8e602c3c-f7fd-433c-b667-ec0f0ba3278c","Type":"ContainerDied","Data":"9dfac37137eb6ecd6da362fb6312a27ee1a0b3a94000ea2be62b621448832cde"}
Jul 15 10:59:14.416058 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:14.415830    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"
Jul 15 10:59:14.416058 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:14.415872    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc" event={"ID":"8e602c3c-f7fd-433c-b667-ec0f0ba3278c","Type":"ContainerDied","Data":"25d605163f340202f850641eb4d3079f37f539d8949645de81e9744a56f494fa"}
Jul 15 10:59:14.416058 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:14.415900    2566 scope.go:117] "RemoveContainer" containerID="9dfac37137eb6ecd6da362fb6312a27ee1a0b3a94000ea2be62b621448832cde"
Jul 15 10:59:14.424816 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:14.424800    2566 scope.go:117] "RemoveContainer" containerID="9dfac37137eb6ecd6da362fb6312a27ee1a0b3a94000ea2be62b621448832cde"
Jul 15 10:59:14.425072 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:59:14.425047    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dfac37137eb6ecd6da362fb6312a27ee1a0b3a94000ea2be62b621448832cde\": container with ID starting with 9dfac37137eb6ecd6da362fb6312a27ee1a0b3a94000ea2be62b621448832cde not found: ID does not exist" containerID="9dfac37137eb6ecd6da362fb6312a27ee1a0b3a94000ea2be62b621448832cde"
Jul 15 10:59:14.425122 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:14.425084    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dfac37137eb6ecd6da362fb6312a27ee1a0b3a94000ea2be62b621448832cde"} err="failed to get container status \"9dfac37137eb6ecd6da362fb6312a27ee1a0b3a94000ea2be62b621448832cde\": rpc error: code = NotFound desc = could not find container \"9dfac37137eb6ecd6da362fb6312a27ee1a0b3a94000ea2be62b621448832cde\": container with ID starting with 9dfac37137eb6ecd6da362fb6312a27ee1a0b3a94000ea2be62b621448832cde not found: ID does not exist"
Jul 15 10:59:14.438548 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:14.438525    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"]
Jul 15 10:59:14.441548 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:14.441528    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-p8gmc"]
Jul 15 10:59:15.267081 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:15.267049    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e602c3c-f7fd-433c-b667-ec0f0ba3278c" path="/var/lib/kubelet/pods/8e602c3c-f7fd-433c-b667-ec0f0ba3278c/volumes"
Jul 15 10:59:18.841596 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:18.841564    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"]
Jul 15 10:59:18.842074 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:18.841876    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e602c3c-f7fd-433c-b667-ec0f0ba3278c" containerName="istio-proxy"
Jul 15 10:59:18.842074 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:18.841886    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e602c3c-f7fd-433c-b667-ec0f0ba3278c" containerName="istio-proxy"
Jul 15 10:59:18.842074 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:18.841942    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e602c3c-f7fd-433c-b667-ec0f0ba3278c" containerName="istio-proxy"
Jul 15 10:59:18.845975 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:18.845956    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:18.849487 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:18.849461    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 10:59:18.849628 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:18.849506    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 15 10:59:18.849628 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:18.849511    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 15 10:59:18.849628 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:18.849540    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 15 10:59:18.859562 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:18.859542    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"]
Jul 15 10:59:18.940145 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:18.940115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:18.940145 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:18.940144    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:18.940351 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:18.940171    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:18.940351 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:18.940269    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:18.940351 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:18.940301    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/058e371a-4746-4a9d-bb5f-a0da4ebeff77-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:18.940351 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:18.940320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4q4r\" (UniqueName: \"kubernetes.io/projected/058e371a-4746-4a9d-bb5f-a0da4ebeff77-kube-api-access-k4q4r\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:19.040987 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:19.040959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/058e371a-4746-4a9d-bb5f-a0da4ebeff77-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:19.041180 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:19.040992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4q4r\" (UniqueName: \"kubernetes.io/projected/058e371a-4746-4a9d-bb5f-a0da4ebeff77-kube-api-access-k4q4r\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:19.041180 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:19.041042    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:19.041180 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:19.041064    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:19.041180 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:19.041084    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:19.041180 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:19.041117    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:19.041508 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:19.041481    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:19.041638 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:19.041567    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:19.041638 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:19.041621    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:19.043481 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:19.043461    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:19.043803 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:19.043786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/058e371a-4746-4a9d-bb5f-a0da4ebeff77-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:19.050753 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:19.050725    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4q4r\" (UniqueName: \"kubernetes.io/projected/058e371a-4746-4a9d-bb5f-a0da4ebeff77-kube-api-access-k4q4r\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:19.155743 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:19.155656    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:19.307151 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:19.307119    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"]
Jul 15 10:59:19.310575 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:59:19.310550    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058e371a_4746_4a9d_bb5f_a0da4ebeff77.slice/crio-12a6cd71ffcf9a543e420b55fbf5490ae16de6f88b29e339f26697a34288c8be WatchSource:0}: Error finding container 12a6cd71ffcf9a543e420b55fbf5490ae16de6f88b29e339f26697a34288c8be: Status 404 returned error can't find the container with id 12a6cd71ffcf9a543e420b55fbf5490ae16de6f88b29e339f26697a34288c8be
Jul 15 10:59:19.312626 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:19.312607    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 10:59:19.442903 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:19.442799    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8" event={"ID":"058e371a-4746-4a9d-bb5f-a0da4ebeff77","Type":"ContainerStarted","Data":"12a6cd71ffcf9a543e420b55fbf5490ae16de6f88b29e339f26697a34288c8be"}
Jul 15 10:59:20.448844 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.448806    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8" event={"ID":"058e371a-4746-4a9d-bb5f-a0da4ebeff77","Type":"ContainerStarted","Data":"f0da8011c9186fb2f9dc422355aad5697d6419b72153a0b3f912e8a9a5f89a85"}
Jul 15 10:59:20.467117 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.467070    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8" podStartSLOduration=1.457326815 podStartE2EDuration="2.467056309s" totalImagesPullingTime="1.009729494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:59:20.466231157 +0000 UTC m=+829.696198687" watchObservedRunningTime="2026-07-15 10:59:20.467056309 +0000 UTC m=+829.697023839"
Jul 15 10:59:20.602876 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.602800    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"]
Jul 15 10:59:20.606178 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.606160    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.611302 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.611281    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-95m52\""
Jul 15 10:59:20.611302 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.611289    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 15 10:59:20.621739 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.621716    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"]
Jul 15 10:59:20.654468 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.654437    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.654468 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.654466    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.654631 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.654506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9638d834-596e-499f-91bc-c8b2f899e46b-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.654631 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.654538    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.654631 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.654592    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9638d834-596e-499f-91bc-c8b2f899e46b-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.654741 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.654651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.654741 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.654686    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.654741 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.654705    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wxx4\" (UniqueName: \"kubernetes.io/projected/9638d834-596e-499f-91bc-c8b2f899e46b-kube-api-access-8wxx4\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.654741 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.654722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9638d834-596e-499f-91bc-c8b2f899e46b-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.755208 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.755176    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9638d834-596e-499f-91bc-c8b2f899e46b-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.755208 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.755212    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.755413 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.755234    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9638d834-596e-499f-91bc-c8b2f899e46b-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.755413 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.755351    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.755413 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.755400    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.755596 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.755430    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wxx4\" (UniqueName: \"kubernetes.io/projected/9638d834-596e-499f-91bc-c8b2f899e46b-kube-api-access-8wxx4\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.755596 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.755459    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9638d834-596e-499f-91bc-c8b2f899e46b-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.755596 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.755519    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.755596 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.755548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.755780 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.755700    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-workload-certs\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.755813 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.755790    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-credential-socket\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.756045 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.756023    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-istio-data\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.756166 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.756149    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9638d834-596e-499f-91bc-c8b2f899e46b-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.756442 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.756416    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-workload-socket\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.757835 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.757811    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-istio-envoy\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.757936 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.757894    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9638d834-596e-499f-91bc-c8b2f899e46b-istio-podinfo\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.764227 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.764204    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9638d834-596e-499f-91bc-c8b2f899e46b-istio-token\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.764477 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.764444    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wxx4\" (UniqueName: \"kubernetes.io/projected/9638d834-596e-499f-91bc-c8b2f899e46b-kube-api-access-8wxx4\") pod \"router-gateway-1-openshift-default-6679d9568-vjjpd\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:20.916519 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:20.916430    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:21.052905 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:21.052876    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"]
Jul 15 10:59:21.055265 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:59:21.055239    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9638d834_596e_499f_91bc_c8b2f899e46b.slice/crio-72ce4d6f48ae50e5e6de589f5a53fc109e09a27cf0e512742099b7ecaef00e25 WatchSource:0}: Error finding container 72ce4d6f48ae50e5e6de589f5a53fc109e09a27cf0e512742099b7ecaef00e25: Status 404 returned error can't find the container with id 72ce4d6f48ae50e5e6de589f5a53fc109e09a27cf0e512742099b7ecaef00e25
Jul 15 10:59:21.057325 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:21.057291    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:59:21.057419 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:21.057373    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:59:21.057477 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:21.057421    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:59:21.156591 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:21.156559    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:21.454782 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:21.454745    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd" event={"ID":"9638d834-596e-499f-91bc-c8b2f899e46b","Type":"ContainerStarted","Data":"5188ea653a22e12a482e1a495f720f5130f3151ef518d0e72f485146d7b0bd55"}
Jul 15 10:59:21.455246 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:21.454807    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd" event={"ID":"9638d834-596e-499f-91bc-c8b2f899e46b","Type":"ContainerStarted","Data":"72ce4d6f48ae50e5e6de589f5a53fc109e09a27cf0e512742099b7ecaef00e25"}
Jul 15 10:59:21.479140 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:21.479093    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd" podStartSLOduration=1.479077749 podStartE2EDuration="1.479077749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:59:21.477270205 +0000 UTC m=+830.707237739" watchObservedRunningTime="2026-07-15 10:59:21.479077749 +0000 UTC m=+830.709045279"
Jul 15 10:59:21.917153 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:21.917121    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:22.922028 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:22.921991    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:23.463422 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:23.463387    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:23.464644 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:23.464622    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:29.156041 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:29.156003    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:29.169610 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:29.169586    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:29.509527 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:29.509481    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:33.519814 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:33.519780    2566 generic.go:356] "Generic (PLEG): container finished" podID="9638d834-596e-499f-91bc-c8b2f899e46b" containerID="5188ea653a22e12a482e1a495f720f5130f3151ef518d0e72f485146d7b0bd55" exitCode=0
Jul 15 10:59:33.520228 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:33.519854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd" event={"ID":"9638d834-596e-499f-91bc-c8b2f899e46b","Type":"ContainerDied","Data":"5188ea653a22e12a482e1a495f720f5130f3151ef518d0e72f485146d7b0bd55"}
Jul 15 10:59:33.520271 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:33.520259    2566 scope.go:117] "RemoveContainer" containerID="5188ea653a22e12a482e1a495f720f5130f3151ef518d0e72f485146d7b0bd55"
Jul 15 10:59:33.521160 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:33.521132    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:59:33.521216 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:33.521203    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:59:33.521257 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:33.521237    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:59:34.528812 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:34.526668    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd" event={"ID":"9638d834-596e-499f-91bc-c8b2f899e46b","Type":"ContainerStarted","Data":"537bab47fc6c4a02695488eb58140b34b682fde72556de7e3675788c5dab3224"}
Jul 15 10:59:34.916758 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:34.916660    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:34.917892 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:34.917854    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd" podUID="9638d834-596e-499f-91bc-c8b2f899e46b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Jul 15 10:59:35.917094 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:35.917040    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd" podUID="9638d834-596e-499f-91bc-c8b2f899e46b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Jul 15 10:59:36.917718 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:36.917675    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd" podUID="9638d834-596e-499f-91bc-c8b2f899e46b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Jul 15 10:59:37.917482 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:37.917442    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd" podUID="9638d834-596e-499f-91bc-c8b2f899e46b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Jul 15 10:59:38.374425 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.374388    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"]
Jul 15 10:59:38.380601 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.380577    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.398245 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.398217    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 10:59:38.398746 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.398713    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 15 10:59:38.401858 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.401834    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"]
Jul 15 10:59:38.464171 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.464143    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:38.512139 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.512101    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjlhp\" (UniqueName: \"kubernetes.io/projected/5bc73b84-de3b-4b52-8d04-035bec641925-kube-api-access-rjlhp\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.512334 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.512147    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.512334 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.512245    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bc73b84-de3b-4b52-8d04-035bec641925-tls-certs\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.512334 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.512271    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-dshm\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.512334 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.512295    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.512476 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.512364    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-model-cache\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.512476 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.512398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-home\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.613098 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.613065    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bc73b84-de3b-4b52-8d04-035bec641925-tls-certs\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.613098 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.613099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-dshm\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.613344 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.613120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.613344 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.613144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-model-cache\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.613344 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.613266    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-home\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.613344 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.613341    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjlhp\" (UniqueName: \"kubernetes.io/projected/5bc73b84-de3b-4b52-8d04-035bec641925-kube-api-access-rjlhp\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.613582 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.613383    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.613735 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.613715    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-home\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.613842 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.613823    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.613909 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.613888    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-model-cache\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.613960 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.613915    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.615702 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.615684    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-dshm\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.615979 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.615960    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bc73b84-de3b-4b52-8d04-035bec641925-tls-certs\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.633470 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.633408    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjlhp\" (UniqueName: \"kubernetes.io/projected/5bc73b84-de3b-4b52-8d04-035bec641925-kube-api-access-rjlhp\") pod \"precise-prefix-cache-test-kserve-7f794775fc-bfwqj\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.710091 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.710052    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:38.916220 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.916121    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"]
Jul 15 10:59:38.918336 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:38.918302    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd" podUID="9638d834-596e-499f-91bc-c8b2f899e46b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Jul 15 10:59:38.922084 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:59:38.922035    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc73b84_de3b_4b52_8d04_035bec641925.slice/crio-b98348bbb7a7e678e41090fed6e351df7591e693caf859dd9c515aa17b81c1a5 WatchSource:0}: Error finding container b98348bbb7a7e678e41090fed6e351df7591e693caf859dd9c515aa17b81c1a5: Status 404 returned error can't find the container with id b98348bbb7a7e678e41090fed6e351df7591e693caf859dd9c515aa17b81c1a5
Jul 15 10:59:39.550981 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:39.550942    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj" event={"ID":"5bc73b84-de3b-4b52-8d04-035bec641925","Type":"ContainerStarted","Data":"e299997d57a7e9d6469466f66406270f5ba1adbbb4d07aa3620f9dfb7603e3c6"}
Jul 15 10:59:39.551467 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:39.551019    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj" event={"ID":"5bc73b84-de3b-4b52-8d04-035bec641925","Type":"ContainerStarted","Data":"b98348bbb7a7e678e41090fed6e351df7591e693caf859dd9c515aa17b81c1a5"}
Jul 15 10:59:39.924820 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:39.924724    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:39.925979 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:39.925953    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:41.729651 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:41.729603    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"]
Jul 15 10:59:42.569703 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:42.569661    2566 generic.go:356] "Generic (PLEG): container finished" podID="5bc73b84-de3b-4b52-8d04-035bec641925" containerID="e299997d57a7e9d6469466f66406270f5ba1adbbb4d07aa3620f9dfb7603e3c6" exitCode=0
Jul 15 10:59:42.569906 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:42.569753    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj" event={"ID":"5bc73b84-de3b-4b52-8d04-035bec641925","Type":"ContainerDied","Data":"e299997d57a7e9d6469466f66406270f5ba1adbbb4d07aa3620f9dfb7603e3c6"}
Jul 15 10:59:43.578786 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:43.578743    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj" event={"ID":"5bc73b84-de3b-4b52-8d04-035bec641925","Type":"ContainerStarted","Data":"5a2e8f24969e6aa488feb1c0d7b244add476b75b6eadafbeee608f2311ab3caa"}
Jul 15 10:59:43.603116 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:43.603060    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj" podStartSLOduration=2.497629684 podStartE2EDuration="5.603041264s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.10541158s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:59:43.601767737 +0000 UTC m=+852.831735268" watchObservedRunningTime="2026-07-15 10:59:43.603041264 +0000 UTC m=+852.833008795"
Jul 15 10:59:43.710435 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:43.710403    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:47.261673 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:47.261633    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 15 10:59:47.262058 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:47.261706    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 15 10:59:48.710825 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:48.710799    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:48.729441 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:48.729417    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:49.612294 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:49.612256    2566 generic.go:356] "Generic (PLEG): container finished" podID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerID="ef5f52cd1c7d1f8754c62c9e22919149e412774bc33f8e0d2a59433073ab47b3" exitCode=0
Jul 15 10:59:49.612558 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:49.612357    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" event={"ID":"63aad8c8-2687-4adf-9bab-9c36b0086419","Type":"ContainerDied","Data":"ef5f52cd1c7d1f8754c62c9e22919149e412774bc33f8e0d2a59433073ab47b3"}
Jul 15 10:59:49.612986 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:49.612959    2566 scope.go:117] "RemoveContainer" containerID="ef5f52cd1c7d1f8754c62c9e22919149e412774bc33f8e0d2a59433073ab47b3"
Jul 15 10:59:49.614089 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:49.614057    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:59:49.614175 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:49.614126    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:59:49.614175 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:49.614158    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:59:49.625618 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:49.625581    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 10:59:50.578962 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.578922    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"]
Jul 15 10:59:50.596029 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.595991    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"]
Jul 15 10:59:50.596029 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.596030    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.619455 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.619427    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" event={"ID":"63aad8c8-2687-4adf-9bab-9c36b0086419","Type":"ContainerStarted","Data":"f27a0f4bd53ee0ab22a097017bf96977f419eb0c50ff78680f9b8c16d279111a"}
Jul 15 10:59:50.722591 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.722547    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53263949-66ba-4c6e-81dc-11865576a00c-workload-certs\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.722591 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.722588    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xczh7\" (UniqueName: \"kubernetes.io/projected/53263949-66ba-4c6e-81dc-11865576a00c-kube-api-access-xczh7\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.722844 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.722694    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53263949-66ba-4c6e-81dc-11865576a00c-istio-data\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.722844 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.722741    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53263949-66ba-4c6e-81dc-11865576a00c-credential-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.722844 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.722817    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53263949-66ba-4c6e-81dc-11865576a00c-istio-envoy\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.722965 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.722853    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53263949-66ba-4c6e-81dc-11865576a00c-istio-token\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.722965 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.722880    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53263949-66ba-4c6e-81dc-11865576a00c-workload-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.722965 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.722910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53263949-66ba-4c6e-81dc-11865576a00c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.722965 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.722943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53263949-66ba-4c6e-81dc-11865576a00c-istio-podinfo\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.798186 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.798149    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:59:50.798452 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.798430    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 10:59:50.798530 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.798480    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 10:59:50.823897 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.823866    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53263949-66ba-4c6e-81dc-11865576a00c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.824023 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.823903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53263949-66ba-4c6e-81dc-11865576a00c-istio-podinfo\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.824023 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.823963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53263949-66ba-4c6e-81dc-11865576a00c-workload-certs\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.824023 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.823986    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xczh7\" (UniqueName: \"kubernetes.io/projected/53263949-66ba-4c6e-81dc-11865576a00c-kube-api-access-xczh7\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.824187 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.824029    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53263949-66ba-4c6e-81dc-11865576a00c-istio-data\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.824187 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.824152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53263949-66ba-4c6e-81dc-11865576a00c-credential-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.824291 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.824204    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53263949-66ba-4c6e-81dc-11865576a00c-istio-envoy\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.824291 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.824237    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53263949-66ba-4c6e-81dc-11865576a00c-istio-token\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.824291 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.824263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53263949-66ba-4c6e-81dc-11865576a00c-workload-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.824543 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.824381    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53263949-66ba-4c6e-81dc-11865576a00c-istio-data\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.824543 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.824432    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53263949-66ba-4c6e-81dc-11865576a00c-workload-certs\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.824647 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.824605    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53263949-66ba-4c6e-81dc-11865576a00c-workload-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.824723 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.824703    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53263949-66ba-4c6e-81dc-11865576a00c-credential-socket\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.824793 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.824768    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53263949-66ba-4c6e-81dc-11865576a00c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.826524 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.826485    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53263949-66ba-4c6e-81dc-11865576a00c-istio-envoy\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.826669 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.826652    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53263949-66ba-4c6e-81dc-11865576a00c-istio-podinfo\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.834173 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.834113    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53263949-66ba-4c6e-81dc-11865576a00c-istio-token\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.834348 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.834326    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xczh7\" (UniqueName: \"kubernetes.io/projected/53263949-66ba-4c6e-81dc-11865576a00c-kube-api-access-xczh7\") pod \"router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7\" (UID: \"53263949-66ba-4c6e-81dc-11865576a00c\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:50.908890 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:50.908846    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:51.123594 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:51.123572    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"]
Jul 15 10:59:51.125843 ip-10-0-133-26 kubenswrapper[2566]: W0715 10:59:51.125811    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53263949_66ba_4c6e_81dc_11865576a00c.slice/crio-badc93225505c88f4e89dd0471264d3f7b087e7faebf463d88ef80cecee6149e WatchSource:0}: Error finding container badc93225505c88f4e89dd0471264d3f7b087e7faebf463d88ef80cecee6149e: Status 404 returned error can't find the container with id badc93225505c88f4e89dd0471264d3f7b087e7faebf463d88ef80cecee6149e
Jul 15 10:59:51.127944 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:51.127898    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:59:51.128050 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:51.127988    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:59:51.128050 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:51.128033    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 10:59:51.626763 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:51.626724    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7" event={"ID":"53263949-66ba-4c6e-81dc-11865576a00c","Type":"ContainerStarted","Data":"14554df8c1a8ac784d8a216501396fcae43ffe1f248e470c360550e12e72d5a8"}
Jul 15 10:59:51.627254 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:51.626786    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7" event={"ID":"53263949-66ba-4c6e-81dc-11865576a00c","Type":"ContainerStarted","Data":"badc93225505c88f4e89dd0471264d3f7b087e7faebf463d88ef80cecee6149e"}
Jul 15 10:59:51.669370 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:51.669313    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7" podStartSLOduration=1.669292999 podStartE2EDuration="1.669292999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 10:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 10:59:51.665932602 +0000 UTC m=+860.895900136" watchObservedRunningTime="2026-07-15 10:59:51.669292999 +0000 UTC m=+860.899260611"
Jul 15 10:59:51.798024 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:51.797983    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 10:59:51.798192 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:51.798040    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 10:59:51.909665 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:51.909574    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:52.798184 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:52.798144    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 10:59:52.798595 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:52.798202    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 10:59:52.915329 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:52.915299    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:53.635179 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:53.635149    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:53.636109 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:53.636090    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 10:59:53.693391 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:53.693343    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"]
Jul 15 10:59:53.693904 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:53.693876    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd" podUID="9638d834-596e-499f-91bc-c8b2f899e46b" containerName="istio-proxy" containerID="cri-o://537bab47fc6c4a02695488eb58140b34b682fde72556de7e3675788c5dab3224" gracePeriod=30
Jul 15 10:59:53.798603 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:53.798567    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 10:59:53.799140 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:53.799109    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 10:59:54.797906 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:54.797870    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 10:59:54.798109 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:54.797928    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 10:59:54.927788 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:54.927742    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd" podUID="9638d834-596e-499f-91bc-c8b2f899e46b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 15 10:59:55.801645 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:55.801615    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:59:55.801987 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:55.801967    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:59:55.802595 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:55.802579    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 10:59:56.754457 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:56.754424    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8" podUID="058e371a-4746-4a9d-bb5f-a0da4ebeff77" containerName="main" containerID="cri-o://f0da8011c9186fb2f9dc422355aad5697d6419b72153a0b3f912e8a9a5f89a85" gracePeriod=45
Jul 15 10:59:57.009237 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.009176    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:57.078201 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.078167    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-model-cache\") pod \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") "
Jul 15 10:59:57.078394 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.078238    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-dshm\") pod \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") "
Jul 15 10:59:57.078394 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.078299    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4q4r\" (UniqueName: \"kubernetes.io/projected/058e371a-4746-4a9d-bb5f-a0da4ebeff77-kube-api-access-k4q4r\") pod \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") "
Jul 15 10:59:57.078394 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.078333    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-home\") pod \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") "
Jul 15 10:59:57.078394 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.078358    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-tmp-dir\") pod \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") "
Jul 15 10:59:57.078394 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.078381    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/058e371a-4746-4a9d-bb5f-a0da4ebeff77-tls-certs\") pod \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\" (UID: \"058e371a-4746-4a9d-bb5f-a0da4ebeff77\") "
Jul 15 10:59:57.078641 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.078461    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-model-cache" (OuterVolumeSpecName: "model-cache") pod "058e371a-4746-4a9d-bb5f-a0da4ebeff77" (UID: "058e371a-4746-4a9d-bb5f-a0da4ebeff77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:57.078691 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.078636    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-home" (OuterVolumeSpecName: "home") pod "058e371a-4746-4a9d-bb5f-a0da4ebeff77" (UID: "058e371a-4746-4a9d-bb5f-a0da4ebeff77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:57.078691 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.078656    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "058e371a-4746-4a9d-bb5f-a0da4ebeff77" (UID: "058e371a-4746-4a9d-bb5f-a0da4ebeff77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:57.078787 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.078736    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:57.078787 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.078748    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:57.078787 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.078757    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:57.080610 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.080587    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-dshm" (OuterVolumeSpecName: "dshm") pod "058e371a-4746-4a9d-bb5f-a0da4ebeff77" (UID: "058e371a-4746-4a9d-bb5f-a0da4ebeff77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:57.080709 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.080622    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/058e371a-4746-4a9d-bb5f-a0da4ebeff77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "058e371a-4746-4a9d-bb5f-a0da4ebeff77" (UID: "058e371a-4746-4a9d-bb5f-a0da4ebeff77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 10:59:57.080749 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.080710    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058e371a-4746-4a9d-bb5f-a0da4ebeff77-kube-api-access-k4q4r" (OuterVolumeSpecName: "kube-api-access-k4q4r") pod "058e371a-4746-4a9d-bb5f-a0da4ebeff77" (UID: "058e371a-4746-4a9d-bb5f-a0da4ebeff77"). InnerVolumeSpecName "kube-api-access-k4q4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:59:57.179436 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.179403    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/058e371a-4746-4a9d-bb5f-a0da4ebeff77-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:57.179436 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.179432    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4q4r\" (UniqueName: \"kubernetes.io/projected/058e371a-4746-4a9d-bb5f-a0da4ebeff77-kube-api-access-k4q4r\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:57.179436 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.179444    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/058e371a-4746-4a9d-bb5f-a0da4ebeff77-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:57.652870 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.652773    2566 generic.go:356] "Generic (PLEG): container finished" podID="058e371a-4746-4a9d-bb5f-a0da4ebeff77" containerID="f0da8011c9186fb2f9dc422355aad5697d6419b72153a0b3f912e8a9a5f89a85" exitCode=0
Jul 15 10:59:57.652870 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.652819    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"
Jul 15 10:59:57.652870 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.652858    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8" event={"ID":"058e371a-4746-4a9d-bb5f-a0da4ebeff77","Type":"ContainerDied","Data":"f0da8011c9186fb2f9dc422355aad5697d6419b72153a0b3f912e8a9a5f89a85"}
Jul 15 10:59:57.653153 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.652905    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8" event={"ID":"058e371a-4746-4a9d-bb5f-a0da4ebeff77","Type":"ContainerDied","Data":"12a6cd71ffcf9a543e420b55fbf5490ae16de6f88b29e339f26697a34288c8be"}
Jul 15 10:59:57.653153 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.652919    2566 scope.go:117] "RemoveContainer" containerID="f0da8011c9186fb2f9dc422355aad5697d6419b72153a0b3f912e8a9a5f89a85"
Jul 15 10:59:57.661084 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.661063    2566 scope.go:117] "RemoveContainer" containerID="f0da8011c9186fb2f9dc422355aad5697d6419b72153a0b3f912e8a9a5f89a85"
Jul 15 10:59:57.661317 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:59:57.661300    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0da8011c9186fb2f9dc422355aad5697d6419b72153a0b3f912e8a9a5f89a85\": container with ID starting with f0da8011c9186fb2f9dc422355aad5697d6419b72153a0b3f912e8a9a5f89a85 not found: ID does not exist" containerID="f0da8011c9186fb2f9dc422355aad5697d6419b72153a0b3f912e8a9a5f89a85"
Jul 15 10:59:57.661384 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.661329    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0da8011c9186fb2f9dc422355aad5697d6419b72153a0b3f912e8a9a5f89a85"} err="failed to get container status \"f0da8011c9186fb2f9dc422355aad5697d6419b72153a0b3f912e8a9a5f89a85\": rpc error: code = NotFound desc = could not find container \"f0da8011c9186fb2f9dc422355aad5697d6419b72153a0b3f912e8a9a5f89a85\": container with ID starting with f0da8011c9186fb2f9dc422355aad5697d6419b72153a0b3f912e8a9a5f89a85 not found: ID does not exist"
Jul 15 10:59:57.671526 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.671482    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"]
Jul 15 10:59:57.676365 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:57.676340    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5b6hn8"]
Jul 15 10:59:58.964408 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:58.964349    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:59.095960 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.095929    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9638d834-596e-499f-91bc-c8b2f899e46b-istio-podinfo\") pod \"9638d834-596e-499f-91bc-c8b2f899e46b\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") "
Jul 15 10:59:59.096126 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.095972    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-istio-envoy\") pod \"9638d834-596e-499f-91bc-c8b2f899e46b\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") "
Jul 15 10:59:59.096126 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.096014    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-workload-certs\") pod \"9638d834-596e-499f-91bc-c8b2f899e46b\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") "
Jul 15 10:59:59.096126 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.096030    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9638d834-596e-499f-91bc-c8b2f899e46b-istiod-ca-cert\") pod \"9638d834-596e-499f-91bc-c8b2f899e46b\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") "
Jul 15 10:59:59.096126 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.096051    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-istio-data\") pod \"9638d834-596e-499f-91bc-c8b2f899e46b\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") "
Jul 15 10:59:59.096126 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.096072    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-credential-socket\") pod \"9638d834-596e-499f-91bc-c8b2f899e46b\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") "
Jul 15 10:59:59.096126 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.096091    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-workload-socket\") pod \"9638d834-596e-499f-91bc-c8b2f899e46b\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") "
Jul 15 10:59:59.096411 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.096168    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wxx4\" (UniqueName: \"kubernetes.io/projected/9638d834-596e-499f-91bc-c8b2f899e46b-kube-api-access-8wxx4\") pod \"9638d834-596e-499f-91bc-c8b2f899e46b\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") "
Jul 15 10:59:59.096411 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.096353    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9638d834-596e-499f-91bc-c8b2f899e46b" (UID: "9638d834-596e-499f-91bc-c8b2f899e46b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:59.096411 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.096380    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9638d834-596e-499f-91bc-c8b2f899e46b" (UID: "9638d834-596e-499f-91bc-c8b2f899e46b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:59.096411 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.096401    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9638d834-596e-499f-91bc-c8b2f899e46b" (UID: "9638d834-596e-499f-91bc-c8b2f899e46b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:59.096665 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.096390    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-istio-data" (OuterVolumeSpecName: "istio-data") pod "9638d834-596e-499f-91bc-c8b2f899e46b" (UID: "9638d834-596e-499f-91bc-c8b2f899e46b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:59.096665 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.096476    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9638d834-596e-499f-91bc-c8b2f899e46b-istio-token\") pod \"9638d834-596e-499f-91bc-c8b2f899e46b\" (UID: \"9638d834-596e-499f-91bc-c8b2f899e46b\") "
Jul 15 10:59:59.097087 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.096540    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9638d834-596e-499f-91bc-c8b2f899e46b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9638d834-596e-499f-91bc-c8b2f899e46b" (UID: "9638d834-596e-499f-91bc-c8b2f899e46b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 10:59:59.097576 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.097555    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-workload-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:59.097767 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.097739    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9638d834-596e-499f-91bc-c8b2f899e46b-istiod-ca-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:59.097767 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.097762    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-istio-data\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:59.097933 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.097775    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-credential-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:59.097933 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.097789    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-workload-socket\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:59.098824 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.098795    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9638d834-596e-499f-91bc-c8b2f899e46b" (UID: "9638d834-596e-499f-91bc-c8b2f899e46b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 10:59:59.099402 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.099379    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9638d834-596e-499f-91bc-c8b2f899e46b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9638d834-596e-499f-91bc-c8b2f899e46b" (UID: "9638d834-596e-499f-91bc-c8b2f899e46b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 15 10:59:59.099402 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.099395    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9638d834-596e-499f-91bc-c8b2f899e46b-kube-api-access-8wxx4" (OuterVolumeSpecName: "kube-api-access-8wxx4") pod "9638d834-596e-499f-91bc-c8b2f899e46b" (UID: "9638d834-596e-499f-91bc-c8b2f899e46b"). InnerVolumeSpecName "kube-api-access-8wxx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:59:59.099567 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.099510    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9638d834-596e-499f-91bc-c8b2f899e46b-istio-token" (OuterVolumeSpecName: "istio-token") pod "9638d834-596e-499f-91bc-c8b2f899e46b" (UID: "9638d834-596e-499f-91bc-c8b2f899e46b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 10:59:59.198744 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.198707    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9638d834-596e-499f-91bc-c8b2f899e46b-istio-token\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:59.198744 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.198741    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9638d834-596e-499f-91bc-c8b2f899e46b-istio-podinfo\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:59.198744 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.198752    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9638d834-596e-499f-91bc-c8b2f899e46b-istio-envoy\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:59.198963 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.198763    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wxx4\" (UniqueName: \"kubernetes.io/projected/9638d834-596e-499f-91bc-c8b2f899e46b-kube-api-access-8wxx4\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 10:59:59.267615 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.267587    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="058e371a-4746-4a9d-bb5f-a0da4ebeff77" path="/var/lib/kubelet/pods/058e371a-4746-4a9d-bb5f-a0da4ebeff77/volumes"
Jul 15 10:59:59.663949 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.663866    2566 generic.go:356] "Generic (PLEG): container finished" podID="9638d834-596e-499f-91bc-c8b2f899e46b" containerID="537bab47fc6c4a02695488eb58140b34b682fde72556de7e3675788c5dab3224" exitCode=0
Jul 15 10:59:59.663949 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.663918    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"
Jul 15 10:59:59.664116 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.663949    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd" event={"ID":"9638d834-596e-499f-91bc-c8b2f899e46b","Type":"ContainerDied","Data":"537bab47fc6c4a02695488eb58140b34b682fde72556de7e3675788c5dab3224"}
Jul 15 10:59:59.664116 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.664011    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd" event={"ID":"9638d834-596e-499f-91bc-c8b2f899e46b","Type":"ContainerDied","Data":"72ce4d6f48ae50e5e6de589f5a53fc109e09a27cf0e512742099b7ecaef00e25"}
Jul 15 10:59:59.664116 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.664034    2566 scope.go:117] "RemoveContainer" containerID="537bab47fc6c4a02695488eb58140b34b682fde72556de7e3675788c5dab3224"
Jul 15 10:59:59.672228 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.672211    2566 scope.go:117] "RemoveContainer" containerID="5188ea653a22e12a482e1a495f720f5130f3151ef518d0e72f485146d7b0bd55"
Jul 15 10:59:59.679746 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.679728    2566 scope.go:117] "RemoveContainer" containerID="537bab47fc6c4a02695488eb58140b34b682fde72556de7e3675788c5dab3224"
Jul 15 10:59:59.679992 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:59:59.679973    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"537bab47fc6c4a02695488eb58140b34b682fde72556de7e3675788c5dab3224\": container with ID starting with 537bab47fc6c4a02695488eb58140b34b682fde72556de7e3675788c5dab3224 not found: ID does not exist" containerID="537bab47fc6c4a02695488eb58140b34b682fde72556de7e3675788c5dab3224"
Jul 15 10:59:59.680122 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.680005    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"537bab47fc6c4a02695488eb58140b34b682fde72556de7e3675788c5dab3224"} err="failed to get container status \"537bab47fc6c4a02695488eb58140b34b682fde72556de7e3675788c5dab3224\": rpc error: code = NotFound desc = could not find container \"537bab47fc6c4a02695488eb58140b34b682fde72556de7e3675788c5dab3224\": container with ID starting with 537bab47fc6c4a02695488eb58140b34b682fde72556de7e3675788c5dab3224 not found: ID does not exist"
Jul 15 10:59:59.680122 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.680029    2566 scope.go:117] "RemoveContainer" containerID="5188ea653a22e12a482e1a495f720f5130f3151ef518d0e72f485146d7b0bd55"
Jul 15 10:59:59.680281 ip-10-0-133-26 kubenswrapper[2566]: E0715 10:59:59.680265    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5188ea653a22e12a482e1a495f720f5130f3151ef518d0e72f485146d7b0bd55\": container with ID starting with 5188ea653a22e12a482e1a495f720f5130f3151ef518d0e72f485146d7b0bd55 not found: ID does not exist" containerID="5188ea653a22e12a482e1a495f720f5130f3151ef518d0e72f485146d7b0bd55"
Jul 15 10:59:59.680332 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.680291    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5188ea653a22e12a482e1a495f720f5130f3151ef518d0e72f485146d7b0bd55"} err="failed to get container status \"5188ea653a22e12a482e1a495f720f5130f3151ef518d0e72f485146d7b0bd55\": rpc error: code = NotFound desc = could not find container \"5188ea653a22e12a482e1a495f720f5130f3151ef518d0e72f485146d7b0bd55\": container with ID starting with 5188ea653a22e12a482e1a495f720f5130f3151ef518d0e72f485146d7b0bd55 not found: ID does not exist"
Jul 15 10:59:59.683656 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.683635    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"]
Jul 15 10:59:59.689110 ip-10-0-133-26 kubenswrapper[2566]: I0715 10:59:59.689089    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-6679d9568-vjjpd"]
Jul 15 11:00:01.267172 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:01.267137    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9638d834-596e-499f-91bc-c8b2f899e46b" path="/var/lib/kubelet/pods/9638d834-596e-499f-91bc-c8b2f899e46b/volumes"
Jul 15 11:00:05.523557 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.523524    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"]
Jul 15 11:00:05.523915 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.523828    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="058e371a-4746-4a9d-bb5f-a0da4ebeff77" containerName="main"
Jul 15 11:00:05.523915 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.523838    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="058e371a-4746-4a9d-bb5f-a0da4ebeff77" containerName="main"
Jul 15 11:00:05.523915 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.523846    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9638d834-596e-499f-91bc-c8b2f899e46b" containerName="istio-proxy"
Jul 15 11:00:05.523915 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.523853    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9638d834-596e-499f-91bc-c8b2f899e46b" containerName="istio-proxy"
Jul 15 11:00:05.523915 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.523908    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="058e371a-4746-4a9d-bb5f-a0da4ebeff77" containerName="main"
Jul 15 11:00:05.523915 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.523916    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9638d834-596e-499f-91bc-c8b2f899e46b" containerName="istio-proxy"
Jul 15 11:00:05.524089 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.523969    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9638d834-596e-499f-91bc-c8b2f899e46b" containerName="istio-proxy"
Jul 15 11:00:05.524089 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.523975    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9638d834-596e-499f-91bc-c8b2f899e46b" containerName="istio-proxy"
Jul 15 11:00:05.524089 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.524028    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9638d834-596e-499f-91bc-c8b2f899e46b" containerName="istio-proxy"
Jul 15 11:00:05.526912 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.526893    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.529468 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.529448    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 15 11:00:05.541467 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.541440    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"]
Jul 15 11:00:05.658181 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.658146    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-kserve-provision-location\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.658181 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.658187    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f85126c9-e82f-44d6-8b35-40b8a85470ba-tls-certs\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.658426 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.658220    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-home\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.658426 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.658239    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x65l\" (UniqueName: \"kubernetes.io/projected/f85126c9-e82f-44d6-8b35-40b8a85470ba-kube-api-access-9x65l\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.658426 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.658312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-tmp-dir\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.658426 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.658369    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-model-cache\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.658426 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.658405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-dshm\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.758863 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.758821    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f85126c9-e82f-44d6-8b35-40b8a85470ba-tls-certs\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.759071 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.758881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-home\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.759071 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.758902    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x65l\" (UniqueName: \"kubernetes.io/projected/f85126c9-e82f-44d6-8b35-40b8a85470ba-kube-api-access-9x65l\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.759071 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.758932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-tmp-dir\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.759071 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.758953    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-model-cache\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.759071 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.758976    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-dshm\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.759331 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.759141    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-kserve-provision-location\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.759421 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.759398    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-model-cache\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.759530 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.759467    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-tmp-dir\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.759667 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.759482    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-home\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.759667 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.759541    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-kserve-provision-location\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.761562 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.761539    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-dshm\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.761662 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.761646    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f85126c9-e82f-44d6-8b35-40b8a85470ba-tls-certs\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.768011 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.767987    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x65l\" (UniqueName: \"kubernetes.io/projected/f85126c9-e82f-44d6-8b35-40b8a85470ba-kube-api-access-9x65l\") pod \"auth-enabled-test-kserve-669d94c795-ksxwd\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.836813 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.836724    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:05.851543 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.851482    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"]
Jul 15 11:00:05.854093 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.854071    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:00:05.856849 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.856800    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-epp-sa-dockercfg-wv5n4\""
Jul 15 11:00:05.867238 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.867209    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"]
Jul 15 11:00:05.961153 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.961121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh572\" (UniqueName: \"kubernetes.io/projected/3d4314fb-596c-4077-a9e6-30c5454d156d-kube-api-access-rh572\") pod \"auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc\" (UID: \"3d4314fb-596c-4077-a9e6-30c5454d156d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:00:05.961326 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.961196    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d4314fb-596c-4077-a9e6-30c5454d156d-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc\" (UID: \"3d4314fb-596c-4077-a9e6-30c5454d156d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:00:05.961326 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.961235    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d4314fb-596c-4077-a9e6-30c5454d156d-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc\" (UID: \"3d4314fb-596c-4077-a9e6-30c5454d156d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:00:05.961326 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:05.961282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d4314fb-596c-4077-a9e6-30c5454d156d-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc\" (UID: \"3d4314fb-596c-4077-a9e6-30c5454d156d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:00:06.002645 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:06.002618    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"]
Jul 15 11:00:06.004942 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:00:06.004915    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85126c9_e82f_44d6_8b35_40b8a85470ba.slice/crio-f09dad2740ccc471fba5224dc37b4449fad3ec40ee456fc367f294b8427baeb8 WatchSource:0}: Error finding container f09dad2740ccc471fba5224dc37b4449fad3ec40ee456fc367f294b8427baeb8: Status 404 returned error can't find the container with id f09dad2740ccc471fba5224dc37b4449fad3ec40ee456fc367f294b8427baeb8
Jul 15 11:00:06.061760 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:06.061736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d4314fb-596c-4077-a9e6-30c5454d156d-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc\" (UID: \"3d4314fb-596c-4077-a9e6-30c5454d156d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:00:06.061844 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:06.061785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh572\" (UniqueName: \"kubernetes.io/projected/3d4314fb-596c-4077-a9e6-30c5454d156d-kube-api-access-rh572\") pod \"auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc\" (UID: \"3d4314fb-596c-4077-a9e6-30c5454d156d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:00:06.061844 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:06.061828    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d4314fb-596c-4077-a9e6-30c5454d156d-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc\" (UID: \"3d4314fb-596c-4077-a9e6-30c5454d156d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:00:06.061938 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:06.061853    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d4314fb-596c-4077-a9e6-30c5454d156d-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc\" (UID: \"3d4314fb-596c-4077-a9e6-30c5454d156d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:00:06.062217 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:06.062198    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d4314fb-596c-4077-a9e6-30c5454d156d-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc\" (UID: \"3d4314fb-596c-4077-a9e6-30c5454d156d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:00:06.062257 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:06.062213    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d4314fb-596c-4077-a9e6-30c5454d156d-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc\" (UID: \"3d4314fb-596c-4077-a9e6-30c5454d156d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:00:06.064394 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:06.064368    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d4314fb-596c-4077-a9e6-30c5454d156d-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc\" (UID: \"3d4314fb-596c-4077-a9e6-30c5454d156d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:00:06.070737 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:06.070713    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh572\" (UniqueName: \"kubernetes.io/projected/3d4314fb-596c-4077-a9e6-30c5454d156d-kube-api-access-rh572\") pod \"auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc\" (UID: \"3d4314fb-596c-4077-a9e6-30c5454d156d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:00:06.167758 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:06.167674    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:00:06.311943 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:06.311910    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"]
Jul 15 11:00:06.315537 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:00:06.315485    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4314fb_596c_4077_a9e6_30c5454d156d.slice/crio-68d6c75cb846eeaa944bfafe8fd135b104102948844dc9819f0217d843444ed9 WatchSource:0}: Error finding container 68d6c75cb846eeaa944bfafe8fd135b104102948844dc9819f0217d843444ed9: Status 404 returned error can't find the container with id 68d6c75cb846eeaa944bfafe8fd135b104102948844dc9819f0217d843444ed9
Jul 15 11:00:06.699811 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:06.699773    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc" event={"ID":"3d4314fb-596c-4077-a9e6-30c5454d156d","Type":"ContainerStarted","Data":"0e588bfb09fb8f29b53da4f630cc05b27d1174875f1fae2fd5dd8a759741c1cb"}
Jul 15 11:00:06.700287 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:06.699836    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc" event={"ID":"3d4314fb-596c-4077-a9e6-30c5454d156d","Type":"ContainerStarted","Data":"68d6c75cb846eeaa944bfafe8fd135b104102948844dc9819f0217d843444ed9"}
Jul 15 11:00:06.701673 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:06.701643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" event={"ID":"f85126c9-e82f-44d6-8b35-40b8a85470ba","Type":"ContainerStarted","Data":"91cfe5d8ebc997e5861d142cc60694dacb8f612187a5565bae07f2336e1979a9"}
Jul 15 11:00:06.701770 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:06.701707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" event={"ID":"f85126c9-e82f-44d6-8b35-40b8a85470ba","Type":"ContainerStarted","Data":"f09dad2740ccc471fba5224dc37b4449fad3ec40ee456fc367f294b8427baeb8"}
Jul 15 11:00:07.716298 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:07.716178    2566 generic.go:356] "Generic (PLEG): container finished" podID="3d4314fb-596c-4077-a9e6-30c5454d156d" containerID="0e588bfb09fb8f29b53da4f630cc05b27d1174875f1fae2fd5dd8a759741c1cb" exitCode=0
Jul 15 11:00:07.716298 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:07.716252    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc" event={"ID":"3d4314fb-596c-4077-a9e6-30c5454d156d","Type":"ContainerDied","Data":"0e588bfb09fb8f29b53da4f630cc05b27d1174875f1fae2fd5dd8a759741c1cb"}
Jul 15 11:00:10.739472 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:10.739433    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc" event={"ID":"3d4314fb-596c-4077-a9e6-30c5454d156d","Type":"ContainerStarted","Data":"6d4396f76e4d35d936be6bd776e665d58f616c470ad22e5b6a97b96a8399b5bf"}
Jul 15 11:00:10.739944 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:10.739555    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:00:10.741144 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:10.741123    2566 generic.go:356] "Generic (PLEG): container finished" podID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerID="91cfe5d8ebc997e5861d142cc60694dacb8f612187a5565bae07f2336e1979a9" exitCode=0
Jul 15 11:00:10.741259 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:10.741152    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" event={"ID":"f85126c9-e82f-44d6-8b35-40b8a85470ba","Type":"ContainerDied","Data":"91cfe5d8ebc997e5861d142cc60694dacb8f612187a5565bae07f2336e1979a9"}
Jul 15 11:00:10.761616 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:10.761564    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc" podStartSLOduration=2.647911884 podStartE2EDuration="5.761548944s" totalImagesPullingTime="2.429061028s" totalInitContainerRuntime="684.576032ms" isStatefulPod=false podCreationTimestamp="2026-07-15 11:00:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:00:10.76102506 +0000 UTC m=+879.990992595" watchObservedRunningTime="2026-07-15 11:00:10.761548944 +0000 UTC m=+879.991516474"
Jul 15 11:00:38.172172 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:38.172143    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:00:38.172710 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:38.172570    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:00:38.897432 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:38.897398    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" event={"ID":"f85126c9-e82f-44d6-8b35-40b8a85470ba","Type":"ContainerStarted","Data":"8d9cc3a8cdbb92d3faec1d04934de2f03a6bf71e596b16c41f5d5dad6d1c1807"}
Jul 15 11:00:38.923946 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:38.923892    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" podStartSLOduration=3.306722593 podStartE2EDuration="33.923876134s" totalImagesPullingTime="27.582543131s" totalInitContainerRuntime="3.03461041s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:00:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:00:38.921133468 +0000 UTC m=+908.151101000" watchObservedRunningTime="2026-07-15 11:00:38.923876134 +0000 UTC m=+908.153843665"
Jul 15 11:00:39.837847 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:39.837808    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:41.751032 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:41.751000    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:00:45.836919 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:45.836876    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:00:45.838378 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:45.838352    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 15 11:00:55.837423 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:00:55.837383    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 15 11:01:05.837878 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:05.837832    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 15 11:01:15.837954 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:15.837896    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 15 11:01:25.837356 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:25.837306    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 15 11:01:26.332087 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:26.332046    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"]
Jul 15 11:01:35.837696 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:35.837653    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 15 11:01:41.353330 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.353265    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj" podUID="5bc73b84-de3b-4b52-8d04-035bec641925" containerName="main" containerID="cri-o://5a2e8f24969e6aa488feb1c0d7b244add476b75b6eadafbeee608f2311ab3caa" gracePeriod=45
Jul 15 11:01:41.608554 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.608481    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 11:01:41.631798 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.631767    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-model-cache\") pod \"5bc73b84-de3b-4b52-8d04-035bec641925\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") "
Jul 15 11:01:41.631895 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.631833    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bc73b84-de3b-4b52-8d04-035bec641925-tls-certs\") pod \"5bc73b84-de3b-4b52-8d04-035bec641925\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") "
Jul 15 11:01:41.631943 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.631910    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-tmp-dir\") pod \"5bc73b84-de3b-4b52-8d04-035bec641925\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") "
Jul 15 11:01:41.632030 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.631953    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-home\") pod \"5bc73b84-de3b-4b52-8d04-035bec641925\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") "
Jul 15 11:01:41.632030 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.631970    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-model-cache" (OuterVolumeSpecName: "model-cache") pod "5bc73b84-de3b-4b52-8d04-035bec641925" (UID: "5bc73b84-de3b-4b52-8d04-035bec641925"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:01:41.632030 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.631989    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-dshm\") pod \"5bc73b84-de3b-4b52-8d04-035bec641925\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") "
Jul 15 11:01:41.632030 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.632020    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-kserve-provision-location\") pod \"5bc73b84-de3b-4b52-8d04-035bec641925\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") "
Jul 15 11:01:41.632239 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.632050    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjlhp\" (UniqueName: \"kubernetes.io/projected/5bc73b84-de3b-4b52-8d04-035bec641925-kube-api-access-rjlhp\") pod \"5bc73b84-de3b-4b52-8d04-035bec641925\" (UID: \"5bc73b84-de3b-4b52-8d04-035bec641925\") "
Jul 15 11:01:41.632239 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.632188    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-home" (OuterVolumeSpecName: "home") pod "5bc73b84-de3b-4b52-8d04-035bec641925" (UID: "5bc73b84-de3b-4b52-8d04-035bec641925"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:01:41.632239 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.632192    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5bc73b84-de3b-4b52-8d04-035bec641925" (UID: "5bc73b84-de3b-4b52-8d04-035bec641925"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:01:41.632384 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.632299    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.632384 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.632314    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.632384 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.632326    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.634439 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.634415    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc73b84-de3b-4b52-8d04-035bec641925-kube-api-access-rjlhp" (OuterVolumeSpecName: "kube-api-access-rjlhp") pod "5bc73b84-de3b-4b52-8d04-035bec641925" (UID: "5bc73b84-de3b-4b52-8d04-035bec641925"). InnerVolumeSpecName "kube-api-access-rjlhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:01:41.634657 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.634636    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-dshm" (OuterVolumeSpecName: "dshm") pod "5bc73b84-de3b-4b52-8d04-035bec641925" (UID: "5bc73b84-de3b-4b52-8d04-035bec641925"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:01:41.634782 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.634764    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bc73b84-de3b-4b52-8d04-035bec641925-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5bc73b84-de3b-4b52-8d04-035bec641925" (UID: "5bc73b84-de3b-4b52-8d04-035bec641925"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:01:41.693745 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.693482    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5bc73b84-de3b-4b52-8d04-035bec641925" (UID: "5bc73b84-de3b-4b52-8d04-035bec641925"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:01:41.733007 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.732966    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bc73b84-de3b-4b52-8d04-035bec641925-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.733007 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.732999    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.733007 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.733011    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bc73b84-de3b-4b52-8d04-035bec641925-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:01:41.733231 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:41.733023    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjlhp\" (UniqueName: \"kubernetes.io/projected/5bc73b84-de3b-4b52-8d04-035bec641925-kube-api-access-rjlhp\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:01:42.215988 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:42.215942    2566 generic.go:356] "Generic (PLEG): container finished" podID="5bc73b84-de3b-4b52-8d04-035bec641925" containerID="5a2e8f24969e6aa488feb1c0d7b244add476b75b6eadafbeee608f2311ab3caa" exitCode=0
Jul 15 11:01:42.216186 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:42.216006    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj" event={"ID":"5bc73b84-de3b-4b52-8d04-035bec641925","Type":"ContainerDied","Data":"5a2e8f24969e6aa488feb1c0d7b244add476b75b6eadafbeee608f2311ab3caa"}
Jul 15 11:01:42.216186 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:42.216028    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"
Jul 15 11:01:42.216186 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:42.216115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj" event={"ID":"5bc73b84-de3b-4b52-8d04-035bec641925","Type":"ContainerDied","Data":"b98348bbb7a7e678e41090fed6e351df7591e693caf859dd9c515aa17b81c1a5"}
Jul 15 11:01:42.216186 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:42.216151    2566 scope.go:117] "RemoveContainer" containerID="5a2e8f24969e6aa488feb1c0d7b244add476b75b6eadafbeee608f2311ab3caa"
Jul 15 11:01:42.227818 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:42.227781    2566 scope.go:117] "RemoveContainer" containerID="e299997d57a7e9d6469466f66406270f5ba1adbbb4d07aa3620f9dfb7603e3c6"
Jul 15 11:01:42.237221 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:42.237204    2566 scope.go:117] "RemoveContainer" containerID="5a2e8f24969e6aa488feb1c0d7b244add476b75b6eadafbeee608f2311ab3caa"
Jul 15 11:01:42.237470 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:01:42.237448    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a2e8f24969e6aa488feb1c0d7b244add476b75b6eadafbeee608f2311ab3caa\": container with ID starting with 5a2e8f24969e6aa488feb1c0d7b244add476b75b6eadafbeee608f2311ab3caa not found: ID does not exist" containerID="5a2e8f24969e6aa488feb1c0d7b244add476b75b6eadafbeee608f2311ab3caa"
Jul 15 11:01:42.237562 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:42.237481    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a2e8f24969e6aa488feb1c0d7b244add476b75b6eadafbeee608f2311ab3caa"} err="failed to get container status \"5a2e8f24969e6aa488feb1c0d7b244add476b75b6eadafbeee608f2311ab3caa\": rpc error: code = NotFound desc = could not find container \"5a2e8f24969e6aa488feb1c0d7b244add476b75b6eadafbeee608f2311ab3caa\": container with ID starting with 5a2e8f24969e6aa488feb1c0d7b244add476b75b6eadafbeee608f2311ab3caa not found: ID does not exist"
Jul 15 11:01:42.237562 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:42.237527    2566 scope.go:117] "RemoveContainer" containerID="e299997d57a7e9d6469466f66406270f5ba1adbbb4d07aa3620f9dfb7603e3c6"
Jul 15 11:01:42.237839 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:01:42.237819    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e299997d57a7e9d6469466f66406270f5ba1adbbb4d07aa3620f9dfb7603e3c6\": container with ID starting with e299997d57a7e9d6469466f66406270f5ba1adbbb4d07aa3620f9dfb7603e3c6 not found: ID does not exist" containerID="e299997d57a7e9d6469466f66406270f5ba1adbbb4d07aa3620f9dfb7603e3c6"
Jul 15 11:01:42.237940 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:42.237842    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e299997d57a7e9d6469466f66406270f5ba1adbbb4d07aa3620f9dfb7603e3c6"} err="failed to get container status \"e299997d57a7e9d6469466f66406270f5ba1adbbb4d07aa3620f9dfb7603e3c6\": rpc error: code = NotFound desc = could not find container \"e299997d57a7e9d6469466f66406270f5ba1adbbb4d07aa3620f9dfb7603e3c6\": container with ID starting with e299997d57a7e9d6469466f66406270f5ba1adbbb4d07aa3620f9dfb7603e3c6 not found: ID does not exist"
Jul 15 11:01:42.241130 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:42.241108    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"]
Jul 15 11:01:42.244579 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:42.244561    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f794775fc-bfwqj"]
Jul 15 11:01:43.267067 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:43.267033    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bc73b84-de3b-4b52-8d04-035bec641925" path="/var/lib/kubelet/pods/5bc73b84-de3b-4b52-8d04-035bec641925/volumes"
Jul 15 11:01:45.837479 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:45.837436    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 15 11:01:55.837633 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:01:55.837537    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 15 11:02:05.837312 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:05.837266    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 15 11:02:15.837760 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:15.837722    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 15 11:02:25.837351 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:25.837312    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 15 11:02:35.847313 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:35.847281    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:02:35.855694 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:35.855671    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:02:40.502063 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:40.502027    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"]
Jul 15 11:02:40.518952 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:40.518921    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"]
Jul 15 11:02:41.750481 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:41.750438    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc" podUID="3d4314fb-596c-4077-a9e6-30c5454d156d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:02:51.750590 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:51.750549    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc" podUID="3d4314fb-596c-4077-a9e6-30c5454d156d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:02:55.523391 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:55.523346    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc" podUID="3d4314fb-596c-4077-a9e6-30c5454d156d" containerName="main" containerID="cri-o://6d4396f76e4d35d936be6bd776e665d58f616c470ad22e5b6a97b96a8399b5bf" gracePeriod=45
Jul 15 11:02:55.545684 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:55.545648    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="main" containerID="cri-o://8d9cc3a8cdbb92d3faec1d04934de2f03a6bf71e596b16c41f5d5dad6d1c1807" gracePeriod=45
Jul 15 11:02:55.773934 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:55.773867    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:02:55.848014 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:55.847973    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Jul 15 11:02:55.874821 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:55.874783    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d4314fb-596c-4077-a9e6-30c5454d156d-tls-certs\") pod \"3d4314fb-596c-4077-a9e6-30c5454d156d\" (UID: \"3d4314fb-596c-4077-a9e6-30c5454d156d\") "
Jul 15 11:02:55.875012 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:55.874843    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh572\" (UniqueName: \"kubernetes.io/projected/3d4314fb-596c-4077-a9e6-30c5454d156d-kube-api-access-rh572\") pod \"3d4314fb-596c-4077-a9e6-30c5454d156d\" (UID: \"3d4314fb-596c-4077-a9e6-30c5454d156d\") "
Jul 15 11:02:55.875012 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:55.874889    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d4314fb-596c-4077-a9e6-30c5454d156d-tokenizer-uds\") pod \"3d4314fb-596c-4077-a9e6-30c5454d156d\" (UID: \"3d4314fb-596c-4077-a9e6-30c5454d156d\") "
Jul 15 11:02:55.875012 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:55.874928    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d4314fb-596c-4077-a9e6-30c5454d156d-kserve-provision-location\") pod \"3d4314fb-596c-4077-a9e6-30c5454d156d\" (UID: \"3d4314fb-596c-4077-a9e6-30c5454d156d\") "
Jul 15 11:02:55.875231 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:55.875201    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4314fb-596c-4077-a9e6-30c5454d156d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3d4314fb-596c-4077-a9e6-30c5454d156d" (UID: "3d4314fb-596c-4077-a9e6-30c5454d156d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:02:55.875787 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:55.875758    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4314fb-596c-4077-a9e6-30c5454d156d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d4314fb-596c-4077-a9e6-30c5454d156d" (UID: "3d4314fb-596c-4077-a9e6-30c5454d156d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:02:55.877183 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:55.877159    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d4314fb-596c-4077-a9e6-30c5454d156d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d4314fb-596c-4077-a9e6-30c5454d156d" (UID: "3d4314fb-596c-4077-a9e6-30c5454d156d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:02:55.877183 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:55.877173    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4314fb-596c-4077-a9e6-30c5454d156d-kube-api-access-rh572" (OuterVolumeSpecName: "kube-api-access-rh572") pod "3d4314fb-596c-4077-a9e6-30c5454d156d" (UID: "3d4314fb-596c-4077-a9e6-30c5454d156d"). InnerVolumeSpecName "kube-api-access-rh572". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:02:55.976518 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:55.976458    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh572\" (UniqueName: \"kubernetes.io/projected/3d4314fb-596c-4077-a9e6-30c5454d156d-kube-api-access-rh572\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:02:55.976518 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:55.976527    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3d4314fb-596c-4077-a9e6-30c5454d156d-tokenizer-uds\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:02:55.976736 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:55.976543    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d4314fb-596c-4077-a9e6-30c5454d156d-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:02:55.976736 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:55.976556    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d4314fb-596c-4077-a9e6-30c5454d156d-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:02:56.286266 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.286241    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:02:56.380281 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.380196    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-model-cache\") pod \"f85126c9-e82f-44d6-8b35-40b8a85470ba\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") "
Jul 15 11:02:56.380281 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.380246    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-tmp-dir\") pod \"f85126c9-e82f-44d6-8b35-40b8a85470ba\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") "
Jul 15 11:02:56.380281 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.380276    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x65l\" (UniqueName: \"kubernetes.io/projected/f85126c9-e82f-44d6-8b35-40b8a85470ba-kube-api-access-9x65l\") pod \"f85126c9-e82f-44d6-8b35-40b8a85470ba\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") "
Jul 15 11:02:56.380593 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.380323    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-home\") pod \"f85126c9-e82f-44d6-8b35-40b8a85470ba\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") "
Jul 15 11:02:56.380593 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.380345    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-kserve-provision-location\") pod \"f85126c9-e82f-44d6-8b35-40b8a85470ba\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") "
Jul 15 11:02:56.380593 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.380362    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-dshm\") pod \"f85126c9-e82f-44d6-8b35-40b8a85470ba\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") "
Jul 15 11:02:56.380593 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.380390    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f85126c9-e82f-44d6-8b35-40b8a85470ba-tls-certs\") pod \"f85126c9-e82f-44d6-8b35-40b8a85470ba\" (UID: \"f85126c9-e82f-44d6-8b35-40b8a85470ba\") "
Jul 15 11:02:56.380808 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.380560    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "f85126c9-e82f-44d6-8b35-40b8a85470ba" (UID: "f85126c9-e82f-44d6-8b35-40b8a85470ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:02:56.380808 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.380725    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:02:56.381387 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.381356    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-home" (OuterVolumeSpecName: "home") pod "f85126c9-e82f-44d6-8b35-40b8a85470ba" (UID: "f85126c9-e82f-44d6-8b35-40b8a85470ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:02:56.383008 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.382975    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-dshm" (OuterVolumeSpecName: "dshm") pod "f85126c9-e82f-44d6-8b35-40b8a85470ba" (UID: "f85126c9-e82f-44d6-8b35-40b8a85470ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:02:56.383299 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.383267    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f85126c9-e82f-44d6-8b35-40b8a85470ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f85126c9-e82f-44d6-8b35-40b8a85470ba" (UID: "f85126c9-e82f-44d6-8b35-40b8a85470ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:02:56.383299 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.383282    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85126c9-e82f-44d6-8b35-40b8a85470ba-kube-api-access-9x65l" (OuterVolumeSpecName: "kube-api-access-9x65l") pod "f85126c9-e82f-44d6-8b35-40b8a85470ba" (UID: "f85126c9-e82f-44d6-8b35-40b8a85470ba"). InnerVolumeSpecName "kube-api-access-9x65l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:02:56.392965 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.392946    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f85126c9-e82f-44d6-8b35-40b8a85470ba" (UID: "f85126c9-e82f-44d6-8b35-40b8a85470ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:02:56.438792 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.438754    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f85126c9-e82f-44d6-8b35-40b8a85470ba" (UID: "f85126c9-e82f-44d6-8b35-40b8a85470ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:02:56.482019 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.481983    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f85126c9-e82f-44d6-8b35-40b8a85470ba-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:02:56.482019 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.482018    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:02:56.482019 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.482028    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x65l\" (UniqueName: \"kubernetes.io/projected/f85126c9-e82f-44d6-8b35-40b8a85470ba-kube-api-access-9x65l\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:02:56.482230 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.482036    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:02:56.482230 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.482045    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:02:56.482230 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.482052    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f85126c9-e82f-44d6-8b35-40b8a85470ba-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:02:56.553162 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.553130    2566 generic.go:356] "Generic (PLEG): container finished" podID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerID="8d9cc3a8cdbb92d3faec1d04934de2f03a6bf71e596b16c41f5d5dad6d1c1807" exitCode=0
Jul 15 11:02:56.553574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.553185    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"
Jul 15 11:02:56.553574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.553221    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" event={"ID":"f85126c9-e82f-44d6-8b35-40b8a85470ba","Type":"ContainerDied","Data":"8d9cc3a8cdbb92d3faec1d04934de2f03a6bf71e596b16c41f5d5dad6d1c1807"}
Jul 15 11:02:56.553574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.553291    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd" event={"ID":"f85126c9-e82f-44d6-8b35-40b8a85470ba","Type":"ContainerDied","Data":"f09dad2740ccc471fba5224dc37b4449fad3ec40ee456fc367f294b8427baeb8"}
Jul 15 11:02:56.553574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.553324    2566 scope.go:117] "RemoveContainer" containerID="8d9cc3a8cdbb92d3faec1d04934de2f03a6bf71e596b16c41f5d5dad6d1c1807"
Jul 15 11:02:56.555109 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.555086    2566 generic.go:356] "Generic (PLEG): container finished" podID="3d4314fb-596c-4077-a9e6-30c5454d156d" containerID="6d4396f76e4d35d936be6bd776e665d58f616c470ad22e5b6a97b96a8399b5bf" exitCode=0
Jul 15 11:02:56.555200 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.555127    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"
Jul 15 11:02:56.555200 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.555170    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc" event={"ID":"3d4314fb-596c-4077-a9e6-30c5454d156d","Type":"ContainerDied","Data":"6d4396f76e4d35d936be6bd776e665d58f616c470ad22e5b6a97b96a8399b5bf"}
Jul 15 11:02:56.555283 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.555210    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc" event={"ID":"3d4314fb-596c-4077-a9e6-30c5454d156d","Type":"ContainerDied","Data":"68d6c75cb846eeaa944bfafe8fd135b104102948844dc9819f0217d843444ed9"}
Jul 15 11:02:56.563299 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.563272    2566 scope.go:117] "RemoveContainer" containerID="91cfe5d8ebc997e5861d142cc60694dacb8f612187a5565bae07f2336e1979a9"
Jul 15 11:02:56.571269 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.571251    2566 scope.go:117] "RemoveContainer" containerID="8d9cc3a8cdbb92d3faec1d04934de2f03a6bf71e596b16c41f5d5dad6d1c1807"
Jul 15 11:02:56.571560 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:02:56.571540    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d9cc3a8cdbb92d3faec1d04934de2f03a6bf71e596b16c41f5d5dad6d1c1807\": container with ID starting with 8d9cc3a8cdbb92d3faec1d04934de2f03a6bf71e596b16c41f5d5dad6d1c1807 not found: ID does not exist" containerID="8d9cc3a8cdbb92d3faec1d04934de2f03a6bf71e596b16c41f5d5dad6d1c1807"
Jul 15 11:02:56.571622 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.571569    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d9cc3a8cdbb92d3faec1d04934de2f03a6bf71e596b16c41f5d5dad6d1c1807"} err="failed to get container status \"8d9cc3a8cdbb92d3faec1d04934de2f03a6bf71e596b16c41f5d5dad6d1c1807\": rpc error: code = NotFound desc = could not find container \"8d9cc3a8cdbb92d3faec1d04934de2f03a6bf71e596b16c41f5d5dad6d1c1807\": container with ID starting with 8d9cc3a8cdbb92d3faec1d04934de2f03a6bf71e596b16c41f5d5dad6d1c1807 not found: ID does not exist"
Jul 15 11:02:56.571622 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.571586    2566 scope.go:117] "RemoveContainer" containerID="91cfe5d8ebc997e5861d142cc60694dacb8f612187a5565bae07f2336e1979a9"
Jul 15 11:02:56.571816 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:02:56.571798    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91cfe5d8ebc997e5861d142cc60694dacb8f612187a5565bae07f2336e1979a9\": container with ID starting with 91cfe5d8ebc997e5861d142cc60694dacb8f612187a5565bae07f2336e1979a9 not found: ID does not exist" containerID="91cfe5d8ebc997e5861d142cc60694dacb8f612187a5565bae07f2336e1979a9"
Jul 15 11:02:56.571859 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.571822    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91cfe5d8ebc997e5861d142cc60694dacb8f612187a5565bae07f2336e1979a9"} err="failed to get container status \"91cfe5d8ebc997e5861d142cc60694dacb8f612187a5565bae07f2336e1979a9\": rpc error: code = NotFound desc = could not find container \"91cfe5d8ebc997e5861d142cc60694dacb8f612187a5565bae07f2336e1979a9\": container with ID starting with 91cfe5d8ebc997e5861d142cc60694dacb8f612187a5565bae07f2336e1979a9 not found: ID does not exist"
Jul 15 11:02:56.571859 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.571837    2566 scope.go:117] "RemoveContainer" containerID="6d4396f76e4d35d936be6bd776e665d58f616c470ad22e5b6a97b96a8399b5bf"
Jul 15 11:02:56.578813 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.578791    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"]
Jul 15 11:02:56.579441 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.579426    2566 scope.go:117] "RemoveContainer" containerID="0e588bfb09fb8f29b53da4f630cc05b27d1174875f1fae2fd5dd8a759741c1cb"
Jul 15 11:02:56.587814 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.587771    2566 scope.go:117] "RemoveContainer" containerID="6d4396f76e4d35d936be6bd776e665d58f616c470ad22e5b6a97b96a8399b5bf"
Jul 15 11:02:56.588079 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:02:56.588061    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d4396f76e4d35d936be6bd776e665d58f616c470ad22e5b6a97b96a8399b5bf\": container with ID starting with 6d4396f76e4d35d936be6bd776e665d58f616c470ad22e5b6a97b96a8399b5bf not found: ID does not exist" containerID="6d4396f76e4d35d936be6bd776e665d58f616c470ad22e5b6a97b96a8399b5bf"
Jul 15 11:02:56.588169 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.588078    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-669d94c795-ksxwd"]
Jul 15 11:02:56.588169 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.588085    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d4396f76e4d35d936be6bd776e665d58f616c470ad22e5b6a97b96a8399b5bf"} err="failed to get container status \"6d4396f76e4d35d936be6bd776e665d58f616c470ad22e5b6a97b96a8399b5bf\": rpc error: code = NotFound desc = could not find container \"6d4396f76e4d35d936be6bd776e665d58f616c470ad22e5b6a97b96a8399b5bf\": container with ID starting with 6d4396f76e4d35d936be6bd776e665d58f616c470ad22e5b6a97b96a8399b5bf not found: ID does not exist"
Jul 15 11:02:56.588169 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.588103    2566 scope.go:117] "RemoveContainer" containerID="0e588bfb09fb8f29b53da4f630cc05b27d1174875f1fae2fd5dd8a759741c1cb"
Jul 15 11:02:56.588345 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:02:56.588328    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e588bfb09fb8f29b53da4f630cc05b27d1174875f1fae2fd5dd8a759741c1cb\": container with ID starting with 0e588bfb09fb8f29b53da4f630cc05b27d1174875f1fae2fd5dd8a759741c1cb not found: ID does not exist" containerID="0e588bfb09fb8f29b53da4f630cc05b27d1174875f1fae2fd5dd8a759741c1cb"
Jul 15 11:02:56.588402 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.588358    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e588bfb09fb8f29b53da4f630cc05b27d1174875f1fae2fd5dd8a759741c1cb"} err="failed to get container status \"0e588bfb09fb8f29b53da4f630cc05b27d1174875f1fae2fd5dd8a759741c1cb\": rpc error: code = NotFound desc = could not find container \"0e588bfb09fb8f29b53da4f630cc05b27d1174875f1fae2fd5dd8a759741c1cb\": container with ID starting with 0e588bfb09fb8f29b53da4f630cc05b27d1174875f1fae2fd5dd8a759741c1cb not found: ID does not exist"
Jul 15 11:02:56.597527 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.597489    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"]
Jul 15 11:02:56.601387 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:56.601365    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7889f97fbd-ljlqc"]
Jul 15 11:02:57.267891 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:57.267859    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d4314fb-596c-4077-a9e6-30c5454d156d" path="/var/lib/kubelet/pods/3d4314fb-596c-4077-a9e6-30c5454d156d/volumes"
Jul 15 11:02:57.268266 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:02:57.268254    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" path="/var/lib/kubelet/pods/f85126c9-e82f-44d6-8b35-40b8a85470ba/volumes"
Jul 15 11:03:22.604091 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.604012    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"]
Jul 15 11:03:22.604460 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.604358    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d4314fb-596c-4077-a9e6-30c5454d156d" containerName="storage-initializer"
Jul 15 11:03:22.604460 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.604370    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d4314fb-596c-4077-a9e6-30c5454d156d" containerName="storage-initializer"
Jul 15 11:03:22.604460 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.604379    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d4314fb-596c-4077-a9e6-30c5454d156d" containerName="main"
Jul 15 11:03:22.604460 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.604384    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d4314fb-596c-4077-a9e6-30c5454d156d" containerName="main"
Jul 15 11:03:22.604460 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.604393    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="main"
Jul 15 11:03:22.604460 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.604398    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="main"
Jul 15 11:03:22.604460 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.604406    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bc73b84-de3b-4b52-8d04-035bec641925" containerName="main"
Jul 15 11:03:22.604460 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.604411    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bc73b84-de3b-4b52-8d04-035bec641925" containerName="main"
Jul 15 11:03:22.604460 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.604425    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="storage-initializer"
Jul 15 11:03:22.604460 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.604430    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="storage-initializer"
Jul 15 11:03:22.604460 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.604441    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bc73b84-de3b-4b52-8d04-035bec641925" containerName="storage-initializer"
Jul 15 11:03:22.604460 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.604446    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bc73b84-de3b-4b52-8d04-035bec641925" containerName="storage-initializer"
Jul 15 11:03:22.604841 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.604508    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bc73b84-de3b-4b52-8d04-035bec641925" containerName="main"
Jul 15 11:03:22.604841 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.604518    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d4314fb-596c-4077-a9e6-30c5454d156d" containerName="main"
Jul 15 11:03:22.604841 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.604524    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f85126c9-e82f-44d6-8b35-40b8a85470ba" containerName="main"
Jul 15 11:03:22.608753 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.608731    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.612545 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.612521    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 11:03:22.612683 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.612613    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 15 11:03:22.612683 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.612650    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 11:03:22.620671 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.620650    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"]
Jul 15 11:03:22.694145 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.694111    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9thv\" (UniqueName: \"kubernetes.io/projected/e9072bcd-9ad1-4427-a90c-901d29d676f0-kube-api-access-k9thv\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.694145 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.694148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.694425 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.694170    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-home\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.694425 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.694192    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-tmp-dir\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.694425 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.694286    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-dshm\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.694425 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.694307    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-model-cache\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.694425 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.694344    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9072bcd-9ad1-4427-a90c-901d29d676f0-tls-certs\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.795574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.795538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9072bcd-9ad1-4427-a90c-901d29d676f0-tls-certs\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.795773 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.795596    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9thv\" (UniqueName: \"kubernetes.io/projected/e9072bcd-9ad1-4427-a90c-901d29d676f0-kube-api-access-k9thv\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.795773 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.795634    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.795773 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.795659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-home\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.795773 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.795686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-tmp-dir\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.795773 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.795749    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-dshm\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.795773 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.795771    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-model-cache\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.796157 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.796135    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-home\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.796253 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.796146    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-model-cache\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.796253 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.796214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-tmp-dir\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.796346 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.796251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.798173 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.798143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-dshm\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.798302 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.798276    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9072bcd-9ad1-4427-a90c-901d29d676f0-tls-certs\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.805920 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.805900    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9thv\" (UniqueName: \"kubernetes.io/projected/e9072bcd-9ad1-4427-a90c-901d29d676f0-kube-api-access-k9thv\") pod \"auth-invalid-token-test-kserve-676866d999-6868f\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.834045 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.834008    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"]
Jul 15 11:03:22.837517 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.837481    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:03:22.840160 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.840139    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-epp-sa-dockercfg-lgsc9\""
Jul 15 11:03:22.847903 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.847881    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"]
Jul 15 11:03:22.896877 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.896797    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e1a1690-4771-4020-9caa-dfa8714dc1c2-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t\" (UID: \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:03:22.896877 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.896838    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e1a1690-4771-4020-9caa-dfa8714dc1c2-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t\" (UID: \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:03:22.896877 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.896863    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8e1a1690-4771-4020-9caa-dfa8714dc1c2-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t\" (UID: \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:03:22.897087 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.896925    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grgmk\" (UniqueName: \"kubernetes.io/projected/8e1a1690-4771-4020-9caa-dfa8714dc1c2-kube-api-access-grgmk\") pod \"auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t\" (UID: \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:03:22.918912 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.918890    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:22.998188 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.998139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e1a1690-4771-4020-9caa-dfa8714dc1c2-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t\" (UID: \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:03:22.998188 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.998182    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e1a1690-4771-4020-9caa-dfa8714dc1c2-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t\" (UID: \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:03:22.998970 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.998204    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8e1a1690-4771-4020-9caa-dfa8714dc1c2-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t\" (UID: \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:03:22.998970 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.998529    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e1a1690-4771-4020-9caa-dfa8714dc1c2-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t\" (UID: \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:03:22.998970 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.998564    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8e1a1690-4771-4020-9caa-dfa8714dc1c2-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t\" (UID: \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:03:22.998970 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:22.998579    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grgmk\" (UniqueName: \"kubernetes.io/projected/8e1a1690-4771-4020-9caa-dfa8714dc1c2-kube-api-access-grgmk\") pod \"auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t\" (UID: \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:03:23.000606 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:23.000583    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e1a1690-4771-4020-9caa-dfa8714dc1c2-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t\" (UID: \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:03:23.008770 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:23.008747    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grgmk\" (UniqueName: \"kubernetes.io/projected/8e1a1690-4771-4020-9caa-dfa8714dc1c2-kube-api-access-grgmk\") pod \"auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t\" (UID: \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:03:23.056729 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:23.056704    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"]
Jul 15 11:03:23.059188 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:03:23.059157    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9072bcd_9ad1_4427_a90c_901d29d676f0.slice/crio-982c34675afadd9ebd31ed67eb55efd856fc73e85bdf36c088b219611a107e21 WatchSource:0}: Error finding container 982c34675afadd9ebd31ed67eb55efd856fc73e85bdf36c088b219611a107e21: Status 404 returned error can't find the container with id 982c34675afadd9ebd31ed67eb55efd856fc73e85bdf36c088b219611a107e21
Jul 15 11:03:23.147576 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:23.147518    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:03:23.293282 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:23.293243    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"]
Jul 15 11:03:23.296358 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:03:23.296332    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1a1690_4771_4020_9caa_dfa8714dc1c2.slice/crio-fff08521553121753399a042ff655d52eb6e3c943bfda2d74c1fce5edbbe8990 WatchSource:0}: Error finding container fff08521553121753399a042ff655d52eb6e3c943bfda2d74c1fce5edbbe8990: Status 404 returned error can't find the container with id fff08521553121753399a042ff655d52eb6e3c943bfda2d74c1fce5edbbe8990
Jul 15 11:03:23.672389 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:23.672356    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t" event={"ID":"8e1a1690-4771-4020-9caa-dfa8714dc1c2","Type":"ContainerStarted","Data":"a5a4c8853b24575711de34ff7b11522c6e8f99680eabb6346a0f4b75464e6506"}
Jul 15 11:03:23.672862 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:23.672415    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t" event={"ID":"8e1a1690-4771-4020-9caa-dfa8714dc1c2","Type":"ContainerStarted","Data":"fff08521553121753399a042ff655d52eb6e3c943bfda2d74c1fce5edbbe8990"}
Jul 15 11:03:23.674124 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:23.674100    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" event={"ID":"e9072bcd-9ad1-4427-a90c-901d29d676f0","Type":"ContainerStarted","Data":"7f75f19eb450f79d4dcc17721708f18a7bf51b0ead7e6321483a0543575c99c8"}
Jul 15 11:03:23.674214 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:23.674137    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" event={"ID":"e9072bcd-9ad1-4427-a90c-901d29d676f0","Type":"ContainerStarted","Data":"982c34675afadd9ebd31ed67eb55efd856fc73e85bdf36c088b219611a107e21"}
Jul 15 11:03:24.681663 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:24.681630    2566 generic.go:356] "Generic (PLEG): container finished" podID="8e1a1690-4771-4020-9caa-dfa8714dc1c2" containerID="a5a4c8853b24575711de34ff7b11522c6e8f99680eabb6346a0f4b75464e6506" exitCode=0
Jul 15 11:03:24.682153 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:24.681718    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t" event={"ID":"8e1a1690-4771-4020-9caa-dfa8714dc1c2","Type":"ContainerDied","Data":"a5a4c8853b24575711de34ff7b11522c6e8f99680eabb6346a0f4b75464e6506"}
Jul 15 11:03:25.690171 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:25.690129    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t" event={"ID":"8e1a1690-4771-4020-9caa-dfa8714dc1c2","Type":"ContainerStarted","Data":"2eb8cc9ed965f0c0252f20eba963b275fb50d719528a41fbe2e514f07c352f3d"}
Jul 15 11:03:25.690648 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:25.690303    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:03:25.709371 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:25.709316    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t" podStartSLOduration=2.999505183 podStartE2EDuration="3.709300186s" totalImagesPullingTime="0s" totalInitContainerRuntime="709.795003ms" isStatefulPod=false podCreationTimestamp="2026-07-15 11:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:03:25.709056818 +0000 UTC m=+1074.939024350" watchObservedRunningTime="2026-07-15 11:03:25.709300186 +0000 UTC m=+1074.939267719"
Jul 15 11:03:26.695996 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:26.695965    2566 generic.go:356] "Generic (PLEG): container finished" podID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerID="7f75f19eb450f79d4dcc17721708f18a7bf51b0ead7e6321483a0543575c99c8" exitCode=0
Jul 15 11:03:26.696382 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:26.696035    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" event={"ID":"e9072bcd-9ad1-4427-a90c-901d29d676f0","Type":"ContainerDied","Data":"7f75f19eb450f79d4dcc17721708f18a7bf51b0ead7e6321483a0543575c99c8"}
Jul 15 11:03:27.702701 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:27.702653    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" event={"ID":"e9072bcd-9ad1-4427-a90c-901d29d676f0","Type":"ContainerStarted","Data":"354d0b268cef726f18070f89d7b5bd98f2061e50b3205665c3b8323fbe6520c7"}
Jul 15 11:03:27.729014 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:27.728953    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" podStartSLOduration=2.784848836 podStartE2EDuration="5.72893838s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.944089544s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:03:27.727111895 +0000 UTC m=+1076.957079418" watchObservedRunningTime="2026-07-15 11:03:27.72893838 +0000 UTC m=+1076.958905911"
Jul 15 11:03:27.919057 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:27.919007    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:32.919193 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:32.919152    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:03:32.920918 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:32.920894    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 15 11:03:42.920123 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:42.920076    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 15 11:03:52.919336 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:52.919293    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 15 11:03:56.698452 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:03:56.698423    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:04:02.919753 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:04:02.919705    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 15 11:04:12.920227 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:04:12.920181    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 15 11:04:22.920340 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:04:22.920299    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 15 11:04:32.920352 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:04:32.920301    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 15 11:04:42.919742 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:04:42.919693    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 15 11:04:52.919409 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:04:52.919306    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 15 11:05:02.919682 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:02.919636    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 15 11:05:12.919482 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:12.919441    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 15 11:05:22.930004 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:22.929971    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:05:22.938272 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:22.938248    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:05:38.198785 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:38.198755    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:05:38.200747 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:38.200723    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:05:39.281905 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:39.281875    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"]
Jul 15 11:05:39.305817 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:39.305786    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"]
Jul 15 11:05:46.698348 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:46.698304    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t" podUID="8e1a1690-4771-4020-9caa-dfa8714dc1c2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:05:54.303346 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:54.303287    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t" podUID="8e1a1690-4771-4020-9caa-dfa8714dc1c2" containerName="main" containerID="cri-o://2eb8cc9ed965f0c0252f20eba963b275fb50d719528a41fbe2e514f07c352f3d" gracePeriod=45
Jul 15 11:05:54.329156 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:54.329119    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="main" containerID="cri-o://354d0b268cef726f18070f89d7b5bd98f2061e50b3205665c3b8323fbe6520c7" gracePeriod=45
Jul 15 11:05:54.553201 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:54.553171    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:05:54.617761 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:54.617655    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8e1a1690-4771-4020-9caa-dfa8714dc1c2-tokenizer-uds\") pod \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\" (UID: \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\") "
Jul 15 11:05:54.617761 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:54.617701    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e1a1690-4771-4020-9caa-dfa8714dc1c2-tls-certs\") pod \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\" (UID: \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\") "
Jul 15 11:05:54.617761 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:54.617727    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e1a1690-4771-4020-9caa-dfa8714dc1c2-kserve-provision-location\") pod \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\" (UID: \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\") "
Jul 15 11:05:54.617761 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:54.617749    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grgmk\" (UniqueName: \"kubernetes.io/projected/8e1a1690-4771-4020-9caa-dfa8714dc1c2-kube-api-access-grgmk\") pod \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\" (UID: \"8e1a1690-4771-4020-9caa-dfa8714dc1c2\") "
Jul 15 11:05:54.618088 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:54.617945    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1a1690-4771-4020-9caa-dfa8714dc1c2-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8e1a1690-4771-4020-9caa-dfa8714dc1c2" (UID: "8e1a1690-4771-4020-9caa-dfa8714dc1c2"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:05:54.618376 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:54.618348    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1a1690-4771-4020-9caa-dfa8714dc1c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e1a1690-4771-4020-9caa-dfa8714dc1c2" (UID: "8e1a1690-4771-4020-9caa-dfa8714dc1c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:05:54.620093 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:54.620062    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1a1690-4771-4020-9caa-dfa8714dc1c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e1a1690-4771-4020-9caa-dfa8714dc1c2" (UID: "8e1a1690-4771-4020-9caa-dfa8714dc1c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:05:54.620093 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:54.620084    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1a1690-4771-4020-9caa-dfa8714dc1c2-kube-api-access-grgmk" (OuterVolumeSpecName: "kube-api-access-grgmk") pod "8e1a1690-4771-4020-9caa-dfa8714dc1c2" (UID: "8e1a1690-4771-4020-9caa-dfa8714dc1c2"). InnerVolumeSpecName "kube-api-access-grgmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:05:54.718354 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:54.718304    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8e1a1690-4771-4020-9caa-dfa8714dc1c2-tokenizer-uds\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:05:54.718354 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:54.718337    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e1a1690-4771-4020-9caa-dfa8714dc1c2-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:05:54.718354 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:54.718346    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e1a1690-4771-4020-9caa-dfa8714dc1c2-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:05:54.718354 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:54.718356    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grgmk\" (UniqueName: \"kubernetes.io/projected/8e1a1690-4771-4020-9caa-dfa8714dc1c2-kube-api-access-grgmk\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:05:54.983702 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:54.983678    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:05:55.020486 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.020458    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-tmp-dir\") pod \"e9072bcd-9ad1-4427-a90c-901d29d676f0\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") "
Jul 15 11:05:55.020677 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.020524    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-kserve-provision-location\") pod \"e9072bcd-9ad1-4427-a90c-901d29d676f0\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") "
Jul 15 11:05:55.020677 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.020569    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-dshm\") pod \"e9072bcd-9ad1-4427-a90c-901d29d676f0\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") "
Jul 15 11:05:55.020677 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.020600    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-model-cache\") pod \"e9072bcd-9ad1-4427-a90c-901d29d676f0\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") "
Jul 15 11:05:55.020677 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.020629    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9thv\" (UniqueName: \"kubernetes.io/projected/e9072bcd-9ad1-4427-a90c-901d29d676f0-kube-api-access-k9thv\") pod \"e9072bcd-9ad1-4427-a90c-901d29d676f0\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") "
Jul 15 11:05:55.020677 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.020656    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-home\") pod \"e9072bcd-9ad1-4427-a90c-901d29d676f0\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") "
Jul 15 11:05:55.020677 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.020677    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9072bcd-9ad1-4427-a90c-901d29d676f0-tls-certs\") pod \"e9072bcd-9ad1-4427-a90c-901d29d676f0\" (UID: \"e9072bcd-9ad1-4427-a90c-901d29d676f0\") "
Jul 15 11:05:55.021215 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.021180    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9072bcd-9ad1-4427-a90c-901d29d676f0" (UID: "e9072bcd-9ad1-4427-a90c-901d29d676f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:05:55.021558 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.021490    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-home" (OuterVolumeSpecName: "home") pod "e9072bcd-9ad1-4427-a90c-901d29d676f0" (UID: "e9072bcd-9ad1-4427-a90c-901d29d676f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:05:55.023564 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.023539    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-dshm" (OuterVolumeSpecName: "dshm") pod "e9072bcd-9ad1-4427-a90c-901d29d676f0" (UID: "e9072bcd-9ad1-4427-a90c-901d29d676f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:05:55.023719 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.023685    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9072bcd-9ad1-4427-a90c-901d29d676f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9072bcd-9ad1-4427-a90c-901d29d676f0" (UID: "e9072bcd-9ad1-4427-a90c-901d29d676f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:05:55.023719 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.023694    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9072bcd-9ad1-4427-a90c-901d29d676f0-kube-api-access-k9thv" (OuterVolumeSpecName: "kube-api-access-k9thv") pod "e9072bcd-9ad1-4427-a90c-901d29d676f0" (UID: "e9072bcd-9ad1-4427-a90c-901d29d676f0"). InnerVolumeSpecName "kube-api-access-k9thv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:05:55.038927 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.038885    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9072bcd-9ad1-4427-a90c-901d29d676f0" (UID: "e9072bcd-9ad1-4427-a90c-901d29d676f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:05:55.086801 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.086741    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e9072bcd-9ad1-4427-a90c-901d29d676f0" (UID: "e9072bcd-9ad1-4427-a90c-901d29d676f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:05:55.122284 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.122252    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:05:55.122284 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.122279    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:05:55.122284 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.122288    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:05:55.122527 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.122298    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9thv\" (UniqueName: \"kubernetes.io/projected/e9072bcd-9ad1-4427-a90c-901d29d676f0-kube-api-access-k9thv\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:05:55.122527 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.122305    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:05:55.122527 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.122313    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9072bcd-9ad1-4427-a90c-901d29d676f0-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:05:55.122527 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.122320    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9072bcd-9ad1-4427-a90c-901d29d676f0-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:05:55.374695 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.374657    2566 generic.go:356] "Generic (PLEG): container finished" podID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerID="354d0b268cef726f18070f89d7b5bd98f2061e50b3205665c3b8323fbe6520c7" exitCode=0
Jul 15 11:05:55.375143 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.374712    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"
Jul 15 11:05:55.375143 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.374744    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" event={"ID":"e9072bcd-9ad1-4427-a90c-901d29d676f0","Type":"ContainerDied","Data":"354d0b268cef726f18070f89d7b5bd98f2061e50b3205665c3b8323fbe6520c7"}
Jul 15 11:05:55.375143 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.374796    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" event={"ID":"e9072bcd-9ad1-4427-a90c-901d29d676f0","Type":"ContainerDied","Data":"982c34675afadd9ebd31ed67eb55efd856fc73e85bdf36c088b219611a107e21"}
Jul 15 11:05:55.375143 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.374822    2566 scope.go:117] "RemoveContainer" containerID="354d0b268cef726f18070f89d7b5bd98f2061e50b3205665c3b8323fbe6520c7"
Jul 15 11:05:55.376868 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.376730    2566 generic.go:356] "Generic (PLEG): container finished" podID="8e1a1690-4771-4020-9caa-dfa8714dc1c2" containerID="2eb8cc9ed965f0c0252f20eba963b275fb50d719528a41fbe2e514f07c352f3d" exitCode=0
Jul 15 11:05:55.376868 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.376782    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"
Jul 15 11:05:55.376868 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.376791    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t" event={"ID":"8e1a1690-4771-4020-9caa-dfa8714dc1c2","Type":"ContainerDied","Data":"2eb8cc9ed965f0c0252f20eba963b275fb50d719528a41fbe2e514f07c352f3d"}
Jul 15 11:05:55.376868 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.376829    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t" event={"ID":"8e1a1690-4771-4020-9caa-dfa8714dc1c2","Type":"ContainerDied","Data":"fff08521553121753399a042ff655d52eb6e3c943bfda2d74c1fce5edbbe8990"}
Jul 15 11:05:55.383568 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.383553    2566 scope.go:117] "RemoveContainer" containerID="7f75f19eb450f79d4dcc17721708f18a7bf51b0ead7e6321483a0543575c99c8"
Jul 15 11:05:55.391559 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.391542    2566 scope.go:117] "RemoveContainer" containerID="354d0b268cef726f18070f89d7b5bd98f2061e50b3205665c3b8323fbe6520c7"
Jul 15 11:05:55.391828 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:05:55.391806    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"354d0b268cef726f18070f89d7b5bd98f2061e50b3205665c3b8323fbe6520c7\": container with ID starting with 354d0b268cef726f18070f89d7b5bd98f2061e50b3205665c3b8323fbe6520c7 not found: ID does not exist" containerID="354d0b268cef726f18070f89d7b5bd98f2061e50b3205665c3b8323fbe6520c7"
Jul 15 11:05:55.391900 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.391838    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"354d0b268cef726f18070f89d7b5bd98f2061e50b3205665c3b8323fbe6520c7"} err="failed to get container status \"354d0b268cef726f18070f89d7b5bd98f2061e50b3205665c3b8323fbe6520c7\": rpc error: code = NotFound desc = could not find container \"354d0b268cef726f18070f89d7b5bd98f2061e50b3205665c3b8323fbe6520c7\": container with ID starting with 354d0b268cef726f18070f89d7b5bd98f2061e50b3205665c3b8323fbe6520c7 not found: ID does not exist"
Jul 15 11:05:55.391900 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.391856    2566 scope.go:117] "RemoveContainer" containerID="7f75f19eb450f79d4dcc17721708f18a7bf51b0ead7e6321483a0543575c99c8"
Jul 15 11:05:55.392105 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:05:55.392088    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f75f19eb450f79d4dcc17721708f18a7bf51b0ead7e6321483a0543575c99c8\": container with ID starting with 7f75f19eb450f79d4dcc17721708f18a7bf51b0ead7e6321483a0543575c99c8 not found: ID does not exist" containerID="7f75f19eb450f79d4dcc17721708f18a7bf51b0ead7e6321483a0543575c99c8"
Jul 15 11:05:55.392166 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.392117    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f75f19eb450f79d4dcc17721708f18a7bf51b0ead7e6321483a0543575c99c8"} err="failed to get container status \"7f75f19eb450f79d4dcc17721708f18a7bf51b0ead7e6321483a0543575c99c8\": rpc error: code = NotFound desc = could not find container \"7f75f19eb450f79d4dcc17721708f18a7bf51b0ead7e6321483a0543575c99c8\": container with ID starting with 7f75f19eb450f79d4dcc17721708f18a7bf51b0ead7e6321483a0543575c99c8 not found: ID does not exist"
Jul 15 11:05:55.392166 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.392139    2566 scope.go:117] "RemoveContainer" containerID="2eb8cc9ed965f0c0252f20eba963b275fb50d719528a41fbe2e514f07c352f3d"
Jul 15 11:05:55.396987 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.396963    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"]
Jul 15 11:05:55.399728 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.399714    2566 scope.go:117] "RemoveContainer" containerID="a5a4c8853b24575711de34ff7b11522c6e8f99680eabb6346a0f4b75464e6506"
Jul 15 11:05:55.405749 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.405730    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f"]
Jul 15 11:05:55.407433 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.407412    2566 scope.go:117] "RemoveContainer" containerID="2eb8cc9ed965f0c0252f20eba963b275fb50d719528a41fbe2e514f07c352f3d"
Jul 15 11:05:55.407705 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:05:55.407686    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eb8cc9ed965f0c0252f20eba963b275fb50d719528a41fbe2e514f07c352f3d\": container with ID starting with 2eb8cc9ed965f0c0252f20eba963b275fb50d719528a41fbe2e514f07c352f3d not found: ID does not exist" containerID="2eb8cc9ed965f0c0252f20eba963b275fb50d719528a41fbe2e514f07c352f3d"
Jul 15 11:05:55.407759 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.407715    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb8cc9ed965f0c0252f20eba963b275fb50d719528a41fbe2e514f07c352f3d"} err="failed to get container status \"2eb8cc9ed965f0c0252f20eba963b275fb50d719528a41fbe2e514f07c352f3d\": rpc error: code = NotFound desc = could not find container \"2eb8cc9ed965f0c0252f20eba963b275fb50d719528a41fbe2e514f07c352f3d\": container with ID starting with 2eb8cc9ed965f0c0252f20eba963b275fb50d719528a41fbe2e514f07c352f3d not found: ID does not exist"
Jul 15 11:05:55.407759 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.407734    2566 scope.go:117] "RemoveContainer" containerID="a5a4c8853b24575711de34ff7b11522c6e8f99680eabb6346a0f4b75464e6506"
Jul 15 11:05:55.407994 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:05:55.407974    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a4c8853b24575711de34ff7b11522c6e8f99680eabb6346a0f4b75464e6506\": container with ID starting with a5a4c8853b24575711de34ff7b11522c6e8f99680eabb6346a0f4b75464e6506 not found: ID does not exist" containerID="a5a4c8853b24575711de34ff7b11522c6e8f99680eabb6346a0f4b75464e6506"
Jul 15 11:05:55.408038 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.408001    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a4c8853b24575711de34ff7b11522c6e8f99680eabb6346a0f4b75464e6506"} err="failed to get container status \"a5a4c8853b24575711de34ff7b11522c6e8f99680eabb6346a0f4b75464e6506\": rpc error: code = NotFound desc = could not find container \"a5a4c8853b24575711de34ff7b11522c6e8f99680eabb6346a0f4b75464e6506\": container with ID starting with a5a4c8853b24575711de34ff7b11522c6e8f99680eabb6346a0f4b75464e6506 not found: ID does not exist"
Jul 15 11:05:55.420056 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.420037    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"]
Jul 15 11:05:55.428284 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.428264    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7bc6974d86ncg8t"]
Jul 15 11:05:55.931619 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:55.931565    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-676866d999-6868f" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 15 11:05:57.266846 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.266812    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e1a1690-4771-4020-9caa-dfa8714dc1c2" path="/var/lib/kubelet/pods/8e1a1690-4771-4020-9caa-dfa8714dc1c2/volumes"
Jul 15 11:05:57.267251 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.267213    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" path="/var/lib/kubelet/pods/e9072bcd-9ad1-4427-a90c-901d29d676f0/volumes"
Jul 15 11:05:57.425723 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.425695    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"]
Jul 15 11:05:57.425993 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.425981    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="storage-initializer"
Jul 15 11:05:57.426037 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.425995    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="storage-initializer"
Jul 15 11:05:57.426037 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.426004    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1a1690-4771-4020-9caa-dfa8714dc1c2" containerName="storage-initializer"
Jul 15 11:05:57.426037 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.426010    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1a1690-4771-4020-9caa-dfa8714dc1c2" containerName="storage-initializer"
Jul 15 11:05:57.426037 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.426015    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1a1690-4771-4020-9caa-dfa8714dc1c2" containerName="main"
Jul 15 11:05:57.426037 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.426020    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1a1690-4771-4020-9caa-dfa8714dc1c2" containerName="main"
Jul 15 11:05:57.426037 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.426035    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="main"
Jul 15 11:05:57.426206 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.426040    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="main"
Jul 15 11:05:57.426206 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.426088    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9072bcd-9ad1-4427-a90c-901d29d676f0" containerName="main"
Jul 15 11:05:57.426206 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.426097    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e1a1690-4771-4020-9caa-dfa8714dc1c2" containerName="main"
Jul 15 11:05:57.429026 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.429005    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.437165 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.437137    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 11:05:57.437270 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.437171    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 15 11:05:57.444315 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.444295    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"]
Jul 15 11:05:57.541116 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.541029    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.541116 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.541061    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.541326 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.541134    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.541326 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.541191    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snt7z\" (UniqueName: \"kubernetes.io/projected/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-kube-api-access-snt7z\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.541326 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.541212    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.541326 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.541243    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.642696 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.642664    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.642696 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.642699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.642923 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.642734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.642923 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.642772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snt7z\" (UniqueName: \"kubernetes.io/projected/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-kube-api-access-snt7z\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.642923 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.642791    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.642923 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.642813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.643166 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.643122    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.643230 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.643185    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.643230 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.643216    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.645124 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.645107    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.645399 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.645380    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.657488 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.657462    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snt7z\" (UniqueName: \"kubernetes.io/projected/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-kube-api-access-snt7z\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.679724 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.679693    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"]
Jul 15 11:05:57.681906 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.681891    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"
Jul 15 11:05:57.684792 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.684770    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-rd47v\""
Jul 15 11:05:57.697176 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.697155    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"]
Jul 15 11:05:57.739469 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.739434    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:05:57.743620 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.743598    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvjr8\" (UniqueName: \"kubernetes.io/projected/5bf75a18-ff5e-4a41-b53c-36971aa48bba-kube-api-access-xvjr8\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v\" (UID: \"5bf75a18-ff5e-4a41-b53c-36971aa48bba\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"
Jul 15 11:05:57.743698 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.743643    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf75a18-ff5e-4a41-b53c-36971aa48bba-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v\" (UID: \"5bf75a18-ff5e-4a41-b53c-36971aa48bba\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"
Jul 15 11:05:57.743698 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.743683    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5bf75a18-ff5e-4a41-b53c-36971aa48bba-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v\" (UID: \"5bf75a18-ff5e-4a41-b53c-36971aa48bba\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"
Jul 15 11:05:57.844847 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.844809    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvjr8\" (UniqueName: \"kubernetes.io/projected/5bf75a18-ff5e-4a41-b53c-36971aa48bba-kube-api-access-xvjr8\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v\" (UID: \"5bf75a18-ff5e-4a41-b53c-36971aa48bba\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"
Jul 15 11:05:57.845001 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.844865    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf75a18-ff5e-4a41-b53c-36971aa48bba-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v\" (UID: \"5bf75a18-ff5e-4a41-b53c-36971aa48bba\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"
Jul 15 11:05:57.845001 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.844905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5bf75a18-ff5e-4a41-b53c-36971aa48bba-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v\" (UID: \"5bf75a18-ff5e-4a41-b53c-36971aa48bba\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"
Jul 15 11:05:57.845448 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.845422    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5bf75a18-ff5e-4a41-b53c-36971aa48bba-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v\" (UID: \"5bf75a18-ff5e-4a41-b53c-36971aa48bba\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"
Jul 15 11:05:57.847723 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.847699    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf75a18-ff5e-4a41-b53c-36971aa48bba-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v\" (UID: \"5bf75a18-ff5e-4a41-b53c-36971aa48bba\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"
Jul 15 11:05:57.855272 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.855246    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvjr8\" (UniqueName: \"kubernetes.io/projected/5bf75a18-ff5e-4a41-b53c-36971aa48bba-kube-api-access-xvjr8\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v\" (UID: \"5bf75a18-ff5e-4a41-b53c-36971aa48bba\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"
Jul 15 11:05:57.872842 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.872820    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"]
Jul 15 11:05:57.874794 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:05:57.874771    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56466f1_0e80_4a3a_aae0_fb225dba8d9b.slice/crio-eb4a32c3ba1a6d6962f1837a3ace7bf188d0a4c18f6721fcb4fb36f79bc1e1c2 WatchSource:0}: Error finding container eb4a32c3ba1a6d6962f1837a3ace7bf188d0a4c18f6721fcb4fb36f79bc1e1c2: Status 404 returned error can't find the container with id eb4a32c3ba1a6d6962f1837a3ace7bf188d0a4c18f6721fcb4fb36f79bc1e1c2
Jul 15 11:05:57.991748 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:57.991719    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"
Jul 15 11:05:58.126553 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:58.126526    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"]
Jul 15 11:05:58.128319 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:05:58.128294    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf75a18_ff5e_4a41_b53c_36971aa48bba.slice/crio-e09cc4c8bab0ca3f5246f90371595bfd2e738503b22cbd56fed1d72d037f0906 WatchSource:0}: Error finding container e09cc4c8bab0ca3f5246f90371595bfd2e738503b22cbd56fed1d72d037f0906: Status 404 returned error can't find the container with id e09cc4c8bab0ca3f5246f90371595bfd2e738503b22cbd56fed1d72d037f0906
Jul 15 11:05:58.395845 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:58.395707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v" event={"ID":"5bf75a18-ff5e-4a41-b53c-36971aa48bba","Type":"ContainerStarted","Data":"1abd485cd13f38c6a4312d9f38f649785f6a1527b3ca5ed1cba6f2f6b4938a34"}
Jul 15 11:05:58.395845 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:58.395782    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"
Jul 15 11:05:58.395845 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:58.395805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v" event={"ID":"5bf75a18-ff5e-4a41-b53c-36971aa48bba","Type":"ContainerStarted","Data":"e09cc4c8bab0ca3f5246f90371595bfd2e738503b22cbd56fed1d72d037f0906"}
Jul 15 11:05:58.397370 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:58.397341    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg" event={"ID":"f56466f1-0e80-4a3a-aae0-fb225dba8d9b","Type":"ContainerStarted","Data":"0cd1bce7ecf76d7ec11221b91acea6ee5d5119843436047b093d5d7e6dc71c14"}
Jul 15 11:05:58.397526 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:58.397393    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg" event={"ID":"f56466f1-0e80-4a3a-aae0-fb225dba8d9b","Type":"ContainerStarted","Data":"eb4a32c3ba1a6d6962f1837a3ace7bf188d0a4c18f6721fcb4fb36f79bc1e1c2"}
Jul 15 11:05:58.414989 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:58.414945    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v" podStartSLOduration=1.414928878 podStartE2EDuration="1.414928878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:05:58.413378339 +0000 UTC m=+1227.643345870" watchObservedRunningTime="2026-07-15 11:05:58.414928878 +0000 UTC m=+1227.644896408"
Jul 15 11:05:58.435875 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:58.435820    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg" podStartSLOduration=1.435804954 podStartE2EDuration="1.435804954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:05:58.434324607 +0000 UTC m=+1227.664292140" watchObservedRunningTime="2026-07-15 11:05:58.435804954 +0000 UTC m=+1227.665772485"
Jul 15 11:05:58.740577 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:05:58.740474    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:06:07.739950 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:07.739913    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:06:07.752351 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:07.752326    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:06:08.450912 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:08.450883    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:06:20.232382 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.232347    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"]
Jul 15 11:06:20.234786 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.234763    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.242625 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.242603    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 11:06:20.242735 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.242627    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 15 11:06:20.251973 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.251943    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"]
Jul 15 11:06:20.335848 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.335813    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53768621-aa81-4e38-a8a9-ea9efb1eeef7-tls-certs\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.336043 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.335852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h66cc\" (UniqueName: \"kubernetes.io/projected/53768621-aa81-4e38-a8a9-ea9efb1eeef7-kube-api-access-h66cc\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.336043 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.336028    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-kserve-provision-location\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.336162 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.336070    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-home\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.336162 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.336131    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-tmp-dir\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.336265 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.336176    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-dshm\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.336265 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.336246    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-model-cache\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.436899 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.436859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-dshm\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.437109 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.436914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-model-cache\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.437109 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.437038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53768621-aa81-4e38-a8a9-ea9efb1eeef7-tls-certs\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.437109 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.437100    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h66cc\" (UniqueName: \"kubernetes.io/projected/53768621-aa81-4e38-a8a9-ea9efb1eeef7-kube-api-access-h66cc\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.437273 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.437179    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-kserve-provision-location\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.437273 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.437209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-home\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.437273 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.437246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-tmp-dir\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.437419 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.437270    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-model-cache\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.437636 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.437616    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-kserve-provision-location\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.437727 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.437652    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-home\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.437870 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.437758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-tmp-dir\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.439294 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.439275    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-dshm\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.439627 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.439606    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53768621-aa81-4e38-a8a9-ea9efb1eeef7-tls-certs\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.450241 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.450218    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h66cc\" (UniqueName: \"kubernetes.io/projected/53768621-aa81-4e38-a8a9-ea9efb1eeef7-kube-api-access-h66cc\") pod \"auth-disabled-test-kserve-66c8497cb9-sdh5z\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.551304 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.551270    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:20.691466 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:20.691426    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"]
Jul 15 11:06:20.694629 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:06:20.694604    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53768621_aa81_4e38_a8a9_ea9efb1eeef7.slice/crio-56a9eb4a67916677c9ad0d68b1af88756a1c7183c427b655306ba63c7fc86441 WatchSource:0}: Error finding container 56a9eb4a67916677c9ad0d68b1af88756a1c7183c427b655306ba63c7fc86441: Status 404 returned error can't find the container with id 56a9eb4a67916677c9ad0d68b1af88756a1c7183c427b655306ba63c7fc86441
Jul 15 11:06:21.500181 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:21.500098    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" event={"ID":"53768621-aa81-4e38-a8a9-ea9efb1eeef7","Type":"ContainerStarted","Data":"b13e4e9a581a352c650392ab1f9a7399eb850e93f9960cab376269e551d73d27"}
Jul 15 11:06:21.500181 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:21.500156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" event={"ID":"53768621-aa81-4e38-a8a9-ea9efb1eeef7","Type":"ContainerStarted","Data":"56a9eb4a67916677c9ad0d68b1af88756a1c7183c427b655306ba63c7fc86441"}
Jul 15 11:06:24.521425 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:24.521391    2566 generic.go:356] "Generic (PLEG): container finished" podID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerID="b13e4e9a581a352c650392ab1f9a7399eb850e93f9960cab376269e551d73d27" exitCode=0
Jul 15 11:06:24.521831 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:24.521465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" event={"ID":"53768621-aa81-4e38-a8a9-ea9efb1eeef7","Type":"ContainerDied","Data":"b13e4e9a581a352c650392ab1f9a7399eb850e93f9960cab376269e551d73d27"}
Jul 15 11:06:25.528965 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:25.528927    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" event={"ID":"53768621-aa81-4e38-a8a9-ea9efb1eeef7","Type":"ContainerStarted","Data":"d2f7673f5e1ae658220f02250bf4afd3fa2f0b466dcd667867be6e6e66aa87aa"}
Jul 15 11:06:25.552123 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:25.552086    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:25.557302 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:25.557258    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" podStartSLOduration=2.5901417220000003 podStartE2EDuration="5.557242048s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.967100326s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:06:25.555591162 +0000 UTC m=+1254.785558693" watchObservedRunningTime="2026-07-15 11:06:25.557242048 +0000 UTC m=+1254.787209579"
Jul 15 11:06:29.404320 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:29.404290    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"
Jul 15 11:06:30.552429 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:30.552396    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:06:30.554020 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:30.553993    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 15 11:06:37.842788 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:37.842746    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"]
Jul 15 11:06:37.864510 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:37.864465    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"]
Jul 15 11:06:39.404274 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:39.404239    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v" podUID="5bf75a18-ff5e-4a41-b53c-36971aa48bba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:06:40.552850 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:40.552802    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 15 11:06:49.403558 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:49.403511    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v" podUID="5bf75a18-ff5e-4a41-b53c-36971aa48bba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:06:50.552299 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:50.552255    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 15 11:06:52.863324 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:52.863259    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v" podUID="5bf75a18-ff5e-4a41-b53c-36971aa48bba" containerName="main" containerID="cri-o://1abd485cd13f38c6a4312d9f38f649785f6a1527b3ca5ed1cba6f2f6b4938a34" gracePeriod=45
Jul 15 11:06:52.884812 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:52.884747    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg" podUID="f56466f1-0e80-4a3a-aae0-fb225dba8d9b" containerName="main" containerID="cri-o://0cd1bce7ecf76d7ec11221b91acea6ee5d5119843436047b093d5d7e6dc71c14" gracePeriod=45
Jul 15 11:06:53.140834 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.140808    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"
Jul 15 11:06:53.155722 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.155702    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:06:53.214556 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.214521    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf75a18-ff5e-4a41-b53c-36971aa48bba-tls-certs\") pod \"5bf75a18-ff5e-4a41-b53c-36971aa48bba\" (UID: \"5bf75a18-ff5e-4a41-b53c-36971aa48bba\") "
Jul 15 11:06:53.214727 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.214579    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-model-cache\") pod \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") "
Jul 15 11:06:53.214727 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.214597    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snt7z\" (UniqueName: \"kubernetes.io/projected/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-kube-api-access-snt7z\") pod \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") "
Jul 15 11:06:53.214851 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.214725    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-tls-certs\") pod \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") "
Jul 15 11:06:53.214851 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.214785    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-home\") pod \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") "
Jul 15 11:06:53.214851 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.214818    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5bf75a18-ff5e-4a41-b53c-36971aa48bba-tokenizer-uds\") pod \"5bf75a18-ff5e-4a41-b53c-36971aa48bba\" (UID: \"5bf75a18-ff5e-4a41-b53c-36971aa48bba\") "
Jul 15 11:06:53.214996 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.214839    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f56466f1-0e80-4a3a-aae0-fb225dba8d9b" (UID: "f56466f1-0e80-4a3a-aae0-fb225dba8d9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:06:53.214996 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.214870    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-tmp-dir\") pod \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") "
Jul 15 11:06:53.214996 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.214900    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-dshm\") pod \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\" (UID: \"f56466f1-0e80-4a3a-aae0-fb225dba8d9b\") "
Jul 15 11:06:53.214996 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.214929    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvjr8\" (UniqueName: \"kubernetes.io/projected/5bf75a18-ff5e-4a41-b53c-36971aa48bba-kube-api-access-xvjr8\") pod \"5bf75a18-ff5e-4a41-b53c-36971aa48bba\" (UID: \"5bf75a18-ff5e-4a41-b53c-36971aa48bba\") "
Jul 15 11:06:53.215176 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.215049    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-home" (OuterVolumeSpecName: "home") pod "f56466f1-0e80-4a3a-aae0-fb225dba8d9b" (UID: "f56466f1-0e80-4a3a-aae0-fb225dba8d9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:06:53.215228 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.215182    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:06:53.215228 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.215201    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:06:53.215781 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.215368    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f56466f1-0e80-4a3a-aae0-fb225dba8d9b" (UID: "f56466f1-0e80-4a3a-aae0-fb225dba8d9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:06:53.215781 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.215457    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf75a18-ff5e-4a41-b53c-36971aa48bba-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "5bf75a18-ff5e-4a41-b53c-36971aa48bba" (UID: "5bf75a18-ff5e-4a41-b53c-36971aa48bba"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:06:53.216905 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.216878    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf75a18-ff5e-4a41-b53c-36971aa48bba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5bf75a18-ff5e-4a41-b53c-36971aa48bba" (UID: "5bf75a18-ff5e-4a41-b53c-36971aa48bba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:06:53.217366 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.217338    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-kube-api-access-snt7z" (OuterVolumeSpecName: "kube-api-access-snt7z") pod "f56466f1-0e80-4a3a-aae0-fb225dba8d9b" (UID: "f56466f1-0e80-4a3a-aae0-fb225dba8d9b"). InnerVolumeSpecName "kube-api-access-snt7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:06:53.217457 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.217405    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f56466f1-0e80-4a3a-aae0-fb225dba8d9b" (UID: "f56466f1-0e80-4a3a-aae0-fb225dba8d9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:06:53.217653 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.217626    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-dshm" (OuterVolumeSpecName: "dshm") pod "f56466f1-0e80-4a3a-aae0-fb225dba8d9b" (UID: "f56466f1-0e80-4a3a-aae0-fb225dba8d9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:06:53.217918 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.217894    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf75a18-ff5e-4a41-b53c-36971aa48bba-kube-api-access-xvjr8" (OuterVolumeSpecName: "kube-api-access-xvjr8") pod "5bf75a18-ff5e-4a41-b53c-36971aa48bba" (UID: "5bf75a18-ff5e-4a41-b53c-36971aa48bba"). InnerVolumeSpecName "kube-api-access-xvjr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:06:53.315718 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.315691    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf75a18-ff5e-4a41-b53c-36971aa48bba-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:06:53.315718 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.315716    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snt7z\" (UniqueName: \"kubernetes.io/projected/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-kube-api-access-snt7z\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:06:53.315885 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.315726    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:06:53.315885 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.315737    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5bf75a18-ff5e-4a41-b53c-36971aa48bba-tokenizer-uds\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:06:53.315885 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.315760    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:06:53.315885 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.315768    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f56466f1-0e80-4a3a-aae0-fb225dba8d9b-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:06:53.315885 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.315776    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvjr8\" (UniqueName: \"kubernetes.io/projected/5bf75a18-ff5e-4a41-b53c-36971aa48bba-kube-api-access-xvjr8\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:06:53.666335 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.666304    2566 generic.go:356] "Generic (PLEG): container finished" podID="5bf75a18-ff5e-4a41-b53c-36971aa48bba" containerID="1abd485cd13f38c6a4312d9f38f649785f6a1527b3ca5ed1cba6f2f6b4938a34" exitCode=0
Jul 15 11:06:53.666573 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.666343    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"
Jul 15 11:06:53.666573 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.666393    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v" event={"ID":"5bf75a18-ff5e-4a41-b53c-36971aa48bba","Type":"ContainerDied","Data":"1abd485cd13f38c6a4312d9f38f649785f6a1527b3ca5ed1cba6f2f6b4938a34"}
Jul 15 11:06:53.666573 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.666450    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v" event={"ID":"5bf75a18-ff5e-4a41-b53c-36971aa48bba","Type":"ContainerDied","Data":"e09cc4c8bab0ca3f5246f90371595bfd2e738503b22cbd56fed1d72d037f0906"}
Jul 15 11:06:53.666573 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.666478    2566 scope.go:117] "RemoveContainer" containerID="1abd485cd13f38c6a4312d9f38f649785f6a1527b3ca5ed1cba6f2f6b4938a34"
Jul 15 11:06:53.667829 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.667808    2566 generic.go:356] "Generic (PLEG): container finished" podID="f56466f1-0e80-4a3a-aae0-fb225dba8d9b" containerID="0cd1bce7ecf76d7ec11221b91acea6ee5d5119843436047b093d5d7e6dc71c14" exitCode=0
Jul 15 11:06:53.667935 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.667847    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg" event={"ID":"f56466f1-0e80-4a3a-aae0-fb225dba8d9b","Type":"ContainerDied","Data":"0cd1bce7ecf76d7ec11221b91acea6ee5d5119843436047b093d5d7e6dc71c14"}
Jul 15 11:06:53.667935 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.667874    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"
Jul 15 11:06:53.667935 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.667894    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg" event={"ID":"f56466f1-0e80-4a3a-aae0-fb225dba8d9b","Type":"ContainerDied","Data":"eb4a32c3ba1a6d6962f1837a3ace7bf188d0a4c18f6721fcb4fb36f79bc1e1c2"}
Jul 15 11:06:53.674960 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.674942    2566 scope.go:117] "RemoveContainer" containerID="1abd485cd13f38c6a4312d9f38f649785f6a1527b3ca5ed1cba6f2f6b4938a34"
Jul 15 11:06:53.675293 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:06:53.675266    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1abd485cd13f38c6a4312d9f38f649785f6a1527b3ca5ed1cba6f2f6b4938a34\": container with ID starting with 1abd485cd13f38c6a4312d9f38f649785f6a1527b3ca5ed1cba6f2f6b4938a34 not found: ID does not exist" containerID="1abd485cd13f38c6a4312d9f38f649785f6a1527b3ca5ed1cba6f2f6b4938a34"
Jul 15 11:06:53.675362 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.675307    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1abd485cd13f38c6a4312d9f38f649785f6a1527b3ca5ed1cba6f2f6b4938a34"} err="failed to get container status \"1abd485cd13f38c6a4312d9f38f649785f6a1527b3ca5ed1cba6f2f6b4938a34\": rpc error: code = NotFound desc = could not find container \"1abd485cd13f38c6a4312d9f38f649785f6a1527b3ca5ed1cba6f2f6b4938a34\": container with ID starting with 1abd485cd13f38c6a4312d9f38f649785f6a1527b3ca5ed1cba6f2f6b4938a34 not found: ID does not exist"
Jul 15 11:06:53.675362 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.675331    2566 scope.go:117] "RemoveContainer" containerID="0cd1bce7ecf76d7ec11221b91acea6ee5d5119843436047b093d5d7e6dc71c14"
Jul 15 11:06:53.682792 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.682774    2566 scope.go:117] "RemoveContainer" containerID="0cd1bce7ecf76d7ec11221b91acea6ee5d5119843436047b093d5d7e6dc71c14"
Jul 15 11:06:53.683084 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:06:53.683061    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cd1bce7ecf76d7ec11221b91acea6ee5d5119843436047b093d5d7e6dc71c14\": container with ID starting with 0cd1bce7ecf76d7ec11221b91acea6ee5d5119843436047b093d5d7e6dc71c14 not found: ID does not exist" containerID="0cd1bce7ecf76d7ec11221b91acea6ee5d5119843436047b093d5d7e6dc71c14"
Jul 15 11:06:53.683139 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.683094    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd1bce7ecf76d7ec11221b91acea6ee5d5119843436047b093d5d7e6dc71c14"} err="failed to get container status \"0cd1bce7ecf76d7ec11221b91acea6ee5d5119843436047b093d5d7e6dc71c14\": rpc error: code = NotFound desc = could not find container \"0cd1bce7ecf76d7ec11221b91acea6ee5d5119843436047b093d5d7e6dc71c14\": container with ID starting with 0cd1bce7ecf76d7ec11221b91acea6ee5d5119843436047b093d5d7e6dc71c14 not found: ID does not exist"
Jul 15 11:06:53.688590 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.688568    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"]
Jul 15 11:06:53.691431 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.691410    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fqhstg"]
Jul 15 11:06:53.702027 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.701993    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"]
Jul 15 11:06:53.706598 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:53.706578    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemzz9v"]
Jul 15 11:06:55.269701 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:55.269663    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bf75a18-ff5e-4a41-b53c-36971aa48bba" path="/var/lib/kubelet/pods/5bf75a18-ff5e-4a41-b53c-36971aa48bba/volumes"
Jul 15 11:06:55.270221 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:06:55.270199    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f56466f1-0e80-4a3a-aae0-fb225dba8d9b" path="/var/lib/kubelet/pods/f56466f1-0e80-4a3a-aae0-fb225dba8d9b/volumes"
Jul 15 11:07:00.552715 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:00.552663    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 15 11:07:10.552768 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:10.552724    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 15 11:07:20.552268 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:20.552216    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 15 11:07:25.121581 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.121549    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"]
Jul 15 11:07:25.121957 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.121871    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf75a18-ff5e-4a41-b53c-36971aa48bba" containerName="main"
Jul 15 11:07:25.121957 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.121883    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf75a18-ff5e-4a41-b53c-36971aa48bba" containerName="main"
Jul 15 11:07:25.121957 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.121895    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f56466f1-0e80-4a3a-aae0-fb225dba8d9b" containerName="main"
Jul 15 11:07:25.121957 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.121904    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f56466f1-0e80-4a3a-aae0-fb225dba8d9b" containerName="main"
Jul 15 11:07:25.122089 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.121964    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f56466f1-0e80-4a3a-aae0-fb225dba8d9b" containerName="main"
Jul 15 11:07:25.122089 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.121971    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf75a18-ff5e-4a41-b53c-36971aa48bba" containerName="main"
Jul 15 11:07:25.123992 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.123972    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.126580 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.126556    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 15 11:07:25.126698 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.126560    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 15 11:07:25.140693 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.140664    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"]
Jul 15 11:07:25.305834 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.305803    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.306013 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.305839    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.306013 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.305876    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.306013 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.305896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.306013 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.305926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97wts\" (UniqueName: \"kubernetes.io/projected/353ad4a4-4f1d-47bd-8b79-255761b37858-kube-api-access-97wts\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.306013 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.305948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/353ad4a4-4f1d-47bd-8b79-255761b37858-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.306013 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.305972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.306013 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.305992    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/353ad4a4-4f1d-47bd-8b79-255761b37858-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.407006 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.406914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97wts\" (UniqueName: \"kubernetes.io/projected/353ad4a4-4f1d-47bd-8b79-255761b37858-kube-api-access-97wts\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.407006 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.406965    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/353ad4a4-4f1d-47bd-8b79-255761b37858-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.407252 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.407007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.407252 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.407034    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/353ad4a4-4f1d-47bd-8b79-255761b37858-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.407252 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.407075    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.407252 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.407099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.407252 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.407158    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.407252 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.407181    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.407593 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.407542    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.407663 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.407635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.407709 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.407663    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.407770 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.407753    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.407957 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.407935    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/353ad4a4-4f1d-47bd-8b79-255761b37858-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.409984 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.409956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.410114 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.410096    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/353ad4a4-4f1d-47bd-8b79-255761b37858-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.418393 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.418364    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97wts\" (UniqueName: \"kubernetes.io/projected/353ad4a4-4f1d-47bd-8b79-255761b37858-kube-api-access-97wts\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.434258 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.434235    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:25.582037 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.582007    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"]
Jul 15 11:07:25.583986 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:07:25.583959    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353ad4a4_4f1d_47bd_8b79_255761b37858.slice/crio-82797d7896a734992096011b396d67aa1b35b3291e8178265e5a68da6c051cab WatchSource:0}: Error finding container 82797d7896a734992096011b396d67aa1b35b3291e8178265e5a68da6c051cab: Status 404 returned error can't find the container with id 82797d7896a734992096011b396d67aa1b35b3291e8178265e5a68da6c051cab
Jul 15 11:07:25.822678 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.822638    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp" event={"ID":"353ad4a4-4f1d-47bd-8b79-255761b37858","Type":"ContainerStarted","Data":"ec4e06843059f4491a3a7da1dd998d7d09504025f9199cfd9a22d1794091f342"}
Jul 15 11:07:25.822957 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:25.822726    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp" event={"ID":"353ad4a4-4f1d-47bd-8b79-255761b37858","Type":"ContainerStarted","Data":"82797d7896a734992096011b396d67aa1b35b3291e8178265e5a68da6c051cab"}
Jul 15 11:07:29.846282 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:29.846248    2566 generic.go:356] "Generic (PLEG): container finished" podID="353ad4a4-4f1d-47bd-8b79-255761b37858" containerID="ec4e06843059f4491a3a7da1dd998d7d09504025f9199cfd9a22d1794091f342" exitCode=0
Jul 15 11:07:29.846789 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:29.846308    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp" event={"ID":"353ad4a4-4f1d-47bd-8b79-255761b37858","Type":"ContainerDied","Data":"ec4e06843059f4491a3a7da1dd998d7d09504025f9199cfd9a22d1794091f342"}
Jul 15 11:07:30.552613 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:30.552561    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 15 11:07:30.853243 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:30.853150    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp" event={"ID":"353ad4a4-4f1d-47bd-8b79-255761b37858","Type":"ContainerStarted","Data":"db4cd808c1f3e0a40bf58d12a70f9a073d366de149ccc86ff69d5824c94b700f"}
Jul 15 11:07:30.880130 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:30.880080    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp" podStartSLOduration=2.606831332 podStartE2EDuration="5.880063326s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.273231994s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:07:30.878619266 +0000 UTC m=+1320.108586797" watchObservedRunningTime="2026-07-15 11:07:30.880063326 +0000 UTC m=+1320.110030857"
Jul 15 11:07:31.434798 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:31.434758    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:35.434534 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:35.434477    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:07:35.436076 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:35.436048    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp" podUID="353ad4a4-4f1d-47bd-8b79-255761b37858" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 15 11:07:40.552157 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:40.552094    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 15 11:07:45.435520 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:45.435461    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp" podUID="353ad4a4-4f1d-47bd-8b79-255761b37858" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 15 11:07:50.552284 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:50.552221    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 15 11:07:55.435714 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:07:55.435601    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp" podUID="353ad4a4-4f1d-47bd-8b79-255761b37858" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 15 11:08:00.551786 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:00.551737    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 15 11:08:05.435883 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:05.435831    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp" podUID="353ad4a4-4f1d-47bd-8b79-255761b37858" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 15 11:08:10.552618 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:10.552557    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 15 11:08:15.435175 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:15.435109    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp" podUID="353ad4a4-4f1d-47bd-8b79-255761b37858" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 15 11:08:20.562039 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:20.562008    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:08:20.570467 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:20.570437    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:08:25.146275 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:25.146234    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"]
Jul 15 11:08:25.435021 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:25.434928    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp" podUID="353ad4a4-4f1d-47bd-8b79-255761b37858" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 15 11:08:35.435328 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:35.435257    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp" podUID="353ad4a4-4f1d-47bd-8b79-255761b37858" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 15 11:08:40.171582 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.171536    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="main" containerID="cri-o://d2f7673f5e1ae658220f02250bf4afd3fa2f0b466dcd667867be6e6e66aa87aa" gracePeriod=45
Jul 15 11:08:40.242601 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.242567    2566 generic.go:356] "Generic (PLEG): container finished" podID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerID="f27a0f4bd53ee0ab22a097017bf96977f419eb0c50ff78680f9b8c16d279111a" exitCode=137
Jul 15 11:08:40.242771 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.242607    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" event={"ID":"63aad8c8-2687-4adf-9bab-9c36b0086419","Type":"ContainerDied","Data":"f27a0f4bd53ee0ab22a097017bf96977f419eb0c50ff78680f9b8c16d279111a"}
Jul 15 11:08:40.242771 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.242664    2566 scope.go:117] "RemoveContainer" containerID="ef5f52cd1c7d1f8754c62c9e22919149e412774bc33f8e0d2a59433073ab47b3"
Jul 15 11:08:40.243115 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.243095    2566 scope.go:117] "RemoveContainer" containerID="f27a0f4bd53ee0ab22a097017bf96977f419eb0c50ff78680f9b8c16d279111a"
Jul 15 11:08:40.243364 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:08:40.243338    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-866675cbb8-fmhhm_openshift-ingress(63aad8c8-2687-4adf-9bab-9c36b0086419)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419"
Jul 15 11:08:40.562862 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.562824    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 15 11:08:40.802019 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.801990    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:08:40.924451 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.924417    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:08:40.994971 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.994934    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53768621-aa81-4e38-a8a9-ea9efb1eeef7-tls-certs\") pod \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") "
Jul 15 11:08:40.995143 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.994988    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-home\") pod \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") "
Jul 15 11:08:40.995143 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.995034    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-model-cache\") pod \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") "
Jul 15 11:08:40.995284 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.995194    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h66cc\" (UniqueName: \"kubernetes.io/projected/53768621-aa81-4e38-a8a9-ea9efb1eeef7-kube-api-access-h66cc\") pod \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") "
Jul 15 11:08:40.995284 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.995215    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-model-cache" (OuterVolumeSpecName: "model-cache") pod "53768621-aa81-4e38-a8a9-ea9efb1eeef7" (UID: "53768621-aa81-4e38-a8a9-ea9efb1eeef7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:08:40.995284 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.995237    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-dshm\") pod \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") "
Jul 15 11:08:40.995449 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.995287    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-kserve-provision-location\") pod \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") "
Jul 15 11:08:40.995449 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.995325    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-tmp-dir\") pod \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\" (UID: \"53768621-aa81-4e38-a8a9-ea9efb1eeef7\") "
Jul 15 11:08:40.995652 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.995621    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:08:40.995652 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.995641    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-home" (OuterVolumeSpecName: "home") pod "53768621-aa81-4e38-a8a9-ea9efb1eeef7" (UID: "53768621-aa81-4e38-a8a9-ea9efb1eeef7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:08:40.997463 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.997437    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53768621-aa81-4e38-a8a9-ea9efb1eeef7-kube-api-access-h66cc" (OuterVolumeSpecName: "kube-api-access-h66cc") pod "53768621-aa81-4e38-a8a9-ea9efb1eeef7" (UID: "53768621-aa81-4e38-a8a9-ea9efb1eeef7"). InnerVolumeSpecName "kube-api-access-h66cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:08:40.997700 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.997677    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53768621-aa81-4e38-a8a9-ea9efb1eeef7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53768621-aa81-4e38-a8a9-ea9efb1eeef7" (UID: "53768621-aa81-4e38-a8a9-ea9efb1eeef7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:08:40.997903 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:40.997878    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-dshm" (OuterVolumeSpecName: "dshm") pod "53768621-aa81-4e38-a8a9-ea9efb1eeef7" (UID: "53768621-aa81-4e38-a8a9-ea9efb1eeef7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:08:41.008813 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.008784    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53768621-aa81-4e38-a8a9-ea9efb1eeef7" (UID: "53768621-aa81-4e38-a8a9-ea9efb1eeef7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:08:41.061690 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.061650    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "53768621-aa81-4e38-a8a9-ea9efb1eeef7" (UID: "53768621-aa81-4e38-a8a9-ea9efb1eeef7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:08:41.096983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.096879    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53768621-aa81-4e38-a8a9-ea9efb1eeef7-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:08:41.096983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.096909    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:08:41.096983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.096918    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h66cc\" (UniqueName: \"kubernetes.io/projected/53768621-aa81-4e38-a8a9-ea9efb1eeef7-kube-api-access-h66cc\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:08:41.096983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.096926    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:08:41.096983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.096934    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:08:41.096983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.096944    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53768621-aa81-4e38-a8a9-ea9efb1eeef7-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:08:41.249121 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.249090    2566 scope.go:117] "RemoveContainer" containerID="f27a0f4bd53ee0ab22a097017bf96977f419eb0c50ff78680f9b8c16d279111a"
Jul 15 11:08:41.249644 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:08:41.249346    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-866675cbb8-fmhhm_openshift-ingress(63aad8c8-2687-4adf-9bab-9c36b0086419)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419"
Jul 15 11:08:41.250524 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.250479    2566 generic.go:356] "Generic (PLEG): container finished" podID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerID="d2f7673f5e1ae658220f02250bf4afd3fa2f0b466dcd667867be6e6e66aa87aa" exitCode=0
Jul 15 11:08:41.250651 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.250532    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" event={"ID":"53768621-aa81-4e38-a8a9-ea9efb1eeef7","Type":"ContainerDied","Data":"d2f7673f5e1ae658220f02250bf4afd3fa2f0b466dcd667867be6e6e66aa87aa"}
Jul 15 11:08:41.250651 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.250546    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"
Jul 15 11:08:41.250651 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.250582    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z" event={"ID":"53768621-aa81-4e38-a8a9-ea9efb1eeef7","Type":"ContainerDied","Data":"56a9eb4a67916677c9ad0d68b1af88756a1c7183c427b655306ba63c7fc86441"}
Jul 15 11:08:41.250651 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.250608    2566 scope.go:117] "RemoveContainer" containerID="d2f7673f5e1ae658220f02250bf4afd3fa2f0b466dcd667867be6e6e66aa87aa"
Jul 15 11:08:41.261040 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.261017    2566 scope.go:117] "RemoveContainer" containerID="b13e4e9a581a352c650392ab1f9a7399eb850e93f9960cab376269e551d73d27"
Jul 15 11:08:41.276110 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.276092    2566 scope.go:117] "RemoveContainer" containerID="d2f7673f5e1ae658220f02250bf4afd3fa2f0b466dcd667867be6e6e66aa87aa"
Jul 15 11:08:41.276400 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:08:41.276380    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f7673f5e1ae658220f02250bf4afd3fa2f0b466dcd667867be6e6e66aa87aa\": container with ID starting with d2f7673f5e1ae658220f02250bf4afd3fa2f0b466dcd667867be6e6e66aa87aa not found: ID does not exist" containerID="d2f7673f5e1ae658220f02250bf4afd3fa2f0b466dcd667867be6e6e66aa87aa"
Jul 15 11:08:41.276446 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.276409    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f7673f5e1ae658220f02250bf4afd3fa2f0b466dcd667867be6e6e66aa87aa"} err="failed to get container status \"d2f7673f5e1ae658220f02250bf4afd3fa2f0b466dcd667867be6e6e66aa87aa\": rpc error: code = NotFound desc = could not find container \"d2f7673f5e1ae658220f02250bf4afd3fa2f0b466dcd667867be6e6e66aa87aa\": container with ID starting with d2f7673f5e1ae658220f02250bf4afd3fa2f0b466dcd667867be6e6e66aa87aa not found: ID does not exist"
Jul 15 11:08:41.276446 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.276427    2566 scope.go:117] "RemoveContainer" containerID="b13e4e9a581a352c650392ab1f9a7399eb850e93f9960cab376269e551d73d27"
Jul 15 11:08:41.276688 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:08:41.276672    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b13e4e9a581a352c650392ab1f9a7399eb850e93f9960cab376269e551d73d27\": container with ID starting with b13e4e9a581a352c650392ab1f9a7399eb850e93f9960cab376269e551d73d27 not found: ID does not exist" containerID="b13e4e9a581a352c650392ab1f9a7399eb850e93f9960cab376269e551d73d27"
Jul 15 11:08:41.276731 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.276693    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b13e4e9a581a352c650392ab1f9a7399eb850e93f9960cab376269e551d73d27"} err="failed to get container status \"b13e4e9a581a352c650392ab1f9a7399eb850e93f9960cab376269e551d73d27\": rpc error: code = NotFound desc = could not find container \"b13e4e9a581a352c650392ab1f9a7399eb850e93f9960cab376269e551d73d27\": container with ID starting with b13e4e9a581a352c650392ab1f9a7399eb850e93f9960cab376269e551d73d27 not found: ID does not exist"
Jul 15 11:08:41.286240 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.286219    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"]
Jul 15 11:08:41.292624 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:41.292601    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-66c8497cb9-sdh5z"]
Jul 15 11:08:43.269427 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:43.269384    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" path="/var/lib/kubelet/pods/53768621-aa81-4e38-a8a9-ea9efb1eeef7/volumes"
Jul 15 11:08:45.435468 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:45.435422    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp" podUID="353ad4a4-4f1d-47bd-8b79-255761b37858" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Jul 15 11:08:53.262246 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:53.262213    2566 scope.go:117] "RemoveContainer" containerID="f27a0f4bd53ee0ab22a097017bf96977f419eb0c50ff78680f9b8c16d279111a"
Jul 15 11:08:53.263350 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:53.263316    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:08:53.263424 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:53.263393    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:08:53.263482 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:53.263424    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:08:54.311338 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:54.311299    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" event={"ID":"63aad8c8-2687-4adf-9bab-9c36b0086419","Type":"ContainerStarted","Data":"4eeb58903d01e6aa70228839108c9fd2b05daeb403c607bdd60ab020f92113c5"}
Jul 15 11:08:54.797572 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:54.797534    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:08:54.798520 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:54.798474    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:08:54.798664 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:54.798552    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:08:55.444436 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:55.444405    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:08:55.452105 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:55.452082    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:08:55.798027 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:55.797996    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:08:55.798201 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:55.798049    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:08:55.802155 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:55.802124    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:08:56.797565 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:56.797531    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:08:56.797931 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:56.797591    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:08:57.798444 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:57.798405    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:08:57.798860 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:57.798459    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:08:58.797817 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:58.797783    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:08:58.797999 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:08:58.797842    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:09:00.465552 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:00.465490    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:09:00.466722 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:00.466702    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:09:18.441445 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:18.441413    2566 generic.go:356] "Generic (PLEG): container finished" podID="53263949-66ba-4c6e-81dc-11865576a00c" containerID="14554df8c1a8ac784d8a216501396fcae43ffe1f248e470c360550e12e72d5a8" exitCode=0
Jul 15 11:09:18.441904 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:18.441483    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7" event={"ID":"53263949-66ba-4c6e-81dc-11865576a00c","Type":"ContainerDied","Data":"14554df8c1a8ac784d8a216501396fcae43ffe1f248e470c360550e12e72d5a8"}
Jul 15 11:09:18.441954 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:18.441907    2566 scope.go:117] "RemoveContainer" containerID="14554df8c1a8ac784d8a216501396fcae43ffe1f248e470c360550e12e72d5a8"
Jul 15 11:09:18.442872 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:18.442844    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:09:18.442964 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:18.442907    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:09:18.442964 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:18.442937    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:09:19.448737 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:19.448701    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7" event={"ID":"53263949-66ba-4c6e-81dc-11865576a00c","Type":"ContainerStarted","Data":"6a2c4591e40d24c11625c4296338f3f5a06ed5c8526918a4d769f9be058b17e9"}
Jul 15 11:09:19.909476 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:19.909435    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 11:09:19.910513 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:19.910465    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7" podUID="53263949-66ba-4c6e-81dc-11865576a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Jul 15 11:09:20.909768 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:20.909724    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7" podUID="53263949-66ba-4c6e-81dc-11865576a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Jul 15 11:09:21.910081 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:21.909995    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7" podUID="53263949-66ba-4c6e-81dc-11865576a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Jul 15 11:09:22.910249 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:22.910209    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7" podUID="53263949-66ba-4c6e-81dc-11865576a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Jul 15 11:09:23.635774 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:23.635744    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 11:09:23.909888 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:23.909794    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7" podUID="53263949-66ba-4c6e-81dc-11865576a00c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Jul 15 11:09:24.940765 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:24.940716    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 11:09:24.943007 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:24.942981    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7"
Jul 15 11:09:37.579269 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:37.579233    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"]
Jul 15 11:09:52.603452 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:52.603414    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp" podUID="353ad4a4-4f1d-47bd-8b79-255761b37858" containerName="main" containerID="cri-o://db4cd808c1f3e0a40bf58d12a70f9a073d366de149ccc86ff69d5824c94b700f" gracePeriod=45
Jul 15 11:09:53.265223 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.265184    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:09:53.360408 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.360319    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-tmp-dir\") pod \"353ad4a4-4f1d-47bd-8b79-255761b37858\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") "
Jul 15 11:09:53.360408 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.360356    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97wts\" (UniqueName: \"kubernetes.io/projected/353ad4a4-4f1d-47bd-8b79-255761b37858-kube-api-access-97wts\") pod \"353ad4a4-4f1d-47bd-8b79-255761b37858\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") "
Jul 15 11:09:53.360408 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.360396    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/353ad4a4-4f1d-47bd-8b79-255761b37858-cabundle-cert\") pod \"353ad4a4-4f1d-47bd-8b79-255761b37858\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") "
Jul 15 11:09:53.360747 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.360432    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/353ad4a4-4f1d-47bd-8b79-255761b37858-tls-certs\") pod \"353ad4a4-4f1d-47bd-8b79-255761b37858\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") "
Jul 15 11:09:53.360747 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.360460    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-dshm\") pod \"353ad4a4-4f1d-47bd-8b79-255761b37858\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") "
Jul 15 11:09:53.360747 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.360524    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-home\") pod \"353ad4a4-4f1d-47bd-8b79-255761b37858\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") "
Jul 15 11:09:53.360747 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.360556    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-model-cache\") pod \"353ad4a4-4f1d-47bd-8b79-255761b37858\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") "
Jul 15 11:09:53.360747 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.360581    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-kserve-provision-location\") pod \"353ad4a4-4f1d-47bd-8b79-255761b37858\" (UID: \"353ad4a4-4f1d-47bd-8b79-255761b37858\") "
Jul 15 11:09:53.361013 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.360874    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353ad4a4-4f1d-47bd-8b79-255761b37858-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "353ad4a4-4f1d-47bd-8b79-255761b37858" (UID: "353ad4a4-4f1d-47bd-8b79-255761b37858"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:09:53.361148 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.361076    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-model-cache" (OuterVolumeSpecName: "model-cache") pod "353ad4a4-4f1d-47bd-8b79-255761b37858" (UID: "353ad4a4-4f1d-47bd-8b79-255761b37858"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:09:53.361321 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.361194    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-home" (OuterVolumeSpecName: "home") pod "353ad4a4-4f1d-47bd-8b79-255761b37858" (UID: "353ad4a4-4f1d-47bd-8b79-255761b37858"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:09:53.362941 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.362906    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353ad4a4-4f1d-47bd-8b79-255761b37858-kube-api-access-97wts" (OuterVolumeSpecName: "kube-api-access-97wts") pod "353ad4a4-4f1d-47bd-8b79-255761b37858" (UID: "353ad4a4-4f1d-47bd-8b79-255761b37858"). InnerVolumeSpecName "kube-api-access-97wts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:09:53.363309 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.363276    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/353ad4a4-4f1d-47bd-8b79-255761b37858-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "353ad4a4-4f1d-47bd-8b79-255761b37858" (UID: "353ad4a4-4f1d-47bd-8b79-255761b37858"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:09:53.363309 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.363275    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-dshm" (OuterVolumeSpecName: "dshm") pod "353ad4a4-4f1d-47bd-8b79-255761b37858" (UID: "353ad4a4-4f1d-47bd-8b79-255761b37858"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:09:53.378638 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.378608    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "353ad4a4-4f1d-47bd-8b79-255761b37858" (UID: "353ad4a4-4f1d-47bd-8b79-255761b37858"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:09:53.423994 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.423952    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "353ad4a4-4f1d-47bd-8b79-255761b37858" (UID: "353ad4a4-4f1d-47bd-8b79-255761b37858"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:09:53.462054 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.462016    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/353ad4a4-4f1d-47bd-8b79-255761b37858-cabundle-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:09:53.462054 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.462048    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/353ad4a4-4f1d-47bd-8b79-255761b37858-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:09:53.462054 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.462056    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:09:53.462054 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.462063    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:09:53.462321 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.462070    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:09:53.462321 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.462079    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:09:53.462321 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.462087    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/353ad4a4-4f1d-47bd-8b79-255761b37858-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:09:53.462321 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.462095    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97wts\" (UniqueName: \"kubernetes.io/projected/353ad4a4-4f1d-47bd-8b79-255761b37858-kube-api-access-97wts\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:09:53.618114 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.618019    2566 generic.go:356] "Generic (PLEG): container finished" podID="353ad4a4-4f1d-47bd-8b79-255761b37858" containerID="db4cd808c1f3e0a40bf58d12a70f9a073d366de149ccc86ff69d5824c94b700f" exitCode=0
Jul 15 11:09:53.618114 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.618073    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"
Jul 15 11:09:53.618114 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.618102    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp" event={"ID":"353ad4a4-4f1d-47bd-8b79-255761b37858","Type":"ContainerDied","Data":"db4cd808c1f3e0a40bf58d12a70f9a073d366de149ccc86ff69d5824c94b700f"}
Jul 15 11:09:53.618686 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.618165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp" event={"ID":"353ad4a4-4f1d-47bd-8b79-255761b37858","Type":"ContainerDied","Data":"82797d7896a734992096011b396d67aa1b35b3291e8178265e5a68da6c051cab"}
Jul 15 11:09:53.618686 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.618196    2566 scope.go:117] "RemoveContainer" containerID="db4cd808c1f3e0a40bf58d12a70f9a073d366de149ccc86ff69d5824c94b700f"
Jul 15 11:09:53.627052 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.627028    2566 scope.go:117] "RemoveContainer" containerID="ec4e06843059f4491a3a7da1dd998d7d09504025f9199cfd9a22d1794091f342"
Jul 15 11:09:53.636859 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.636838    2566 scope.go:117] "RemoveContainer" containerID="db4cd808c1f3e0a40bf58d12a70f9a073d366de149ccc86ff69d5824c94b700f"
Jul 15 11:09:53.637159 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:09:53.637139    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4cd808c1f3e0a40bf58d12a70f9a073d366de149ccc86ff69d5824c94b700f\": container with ID starting with db4cd808c1f3e0a40bf58d12a70f9a073d366de149ccc86ff69d5824c94b700f not found: ID does not exist" containerID="db4cd808c1f3e0a40bf58d12a70f9a073d366de149ccc86ff69d5824c94b700f"
Jul 15 11:09:53.637223 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.637168    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4cd808c1f3e0a40bf58d12a70f9a073d366de149ccc86ff69d5824c94b700f"} err="failed to get container status \"db4cd808c1f3e0a40bf58d12a70f9a073d366de149ccc86ff69d5824c94b700f\": rpc error: code = NotFound desc = could not find container \"db4cd808c1f3e0a40bf58d12a70f9a073d366de149ccc86ff69d5824c94b700f\": container with ID starting with db4cd808c1f3e0a40bf58d12a70f9a073d366de149ccc86ff69d5824c94b700f not found: ID does not exist"
Jul 15 11:09:53.637223 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.637187    2566 scope.go:117] "RemoveContainer" containerID="ec4e06843059f4491a3a7da1dd998d7d09504025f9199cfd9a22d1794091f342"
Jul 15 11:09:53.637464 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:09:53.637446    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec4e06843059f4491a3a7da1dd998d7d09504025f9199cfd9a22d1794091f342\": container with ID starting with ec4e06843059f4491a3a7da1dd998d7d09504025f9199cfd9a22d1794091f342 not found: ID does not exist" containerID="ec4e06843059f4491a3a7da1dd998d7d09504025f9199cfd9a22d1794091f342"
Jul 15 11:09:53.637548 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.637473    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec4e06843059f4491a3a7da1dd998d7d09504025f9199cfd9a22d1794091f342"} err="failed to get container status \"ec4e06843059f4491a3a7da1dd998d7d09504025f9199cfd9a22d1794091f342\": rpc error: code = NotFound desc = could not find container \"ec4e06843059f4491a3a7da1dd998d7d09504025f9199cfd9a22d1794091f342\": container with ID starting with ec4e06843059f4491a3a7da1dd998d7d09504025f9199cfd9a22d1794091f342 not found: ID does not exist"
Jul 15 11:09:53.641132 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.641112    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"]
Jul 15 11:09:53.644294 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:53.644272    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-59c5f99dc4-brzzp"]
Jul 15 11:09:55.267843 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:09:55.267806    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="353ad4a4-4f1d-47bd-8b79-255761b37858" path="/var/lib/kubelet/pods/353ad4a4-4f1d-47bd-8b79-255761b37858/volumes"
Jul 15 11:10:03.629791 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.629754    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"]
Jul 15 11:10:03.630239 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.630092    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="353ad4a4-4f1d-47bd-8b79-255761b37858" containerName="storage-initializer"
Jul 15 11:10:03.630239 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.630104    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="353ad4a4-4f1d-47bd-8b79-255761b37858" containerName="storage-initializer"
Jul 15 11:10:03.630239 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.630114    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="storage-initializer"
Jul 15 11:10:03.630239 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.630121    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="storage-initializer"
Jul 15 11:10:03.630239 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.630130    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="main"
Jul 15 11:10:03.630239 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.630137    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="main"
Jul 15 11:10:03.630239 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.630144    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="353ad4a4-4f1d-47bd-8b79-255761b37858" containerName="main"
Jul 15 11:10:03.630239 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.630148    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="353ad4a4-4f1d-47bd-8b79-255761b37858" containerName="main"
Jul 15 11:10:03.630239 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.630200    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="53768621-aa81-4e38-a8a9-ea9efb1eeef7" containerName="main"
Jul 15 11:10:03.630239 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.630208    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="353ad4a4-4f1d-47bd-8b79-255761b37858" containerName="main"
Jul 15 11:10:03.633053 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.633024    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.635602 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.635582    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 11:10:03.635781 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.635688    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 11:10:03.635902 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.635888    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 15 11:10:03.641536 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.641512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.641618 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.641577    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.641618 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.641606    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq6qf\" (UniqueName: \"kubernetes.io/projected/fd0aaa95-f669-4e19-87bf-5d97d9532618-kube-api-access-qq6qf\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.641699 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.641627    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd0aaa95-f669-4e19-87bf-5d97d9532618-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.641734 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.641704    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.641734 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.641728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.641799 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.641749    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.647432 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.647406    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"]
Jul 15 11:10:03.742284 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.742245    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.742528 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.742296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.742528 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.742320    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq6qf\" (UniqueName: \"kubernetes.io/projected/fd0aaa95-f669-4e19-87bf-5d97d9532618-kube-api-access-qq6qf\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.742528 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.742340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd0aaa95-f669-4e19-87bf-5d97d9532618-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.742528 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.742364    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.742528 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.742378    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.742528 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.742401    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.742827 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.742748    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.742887 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.742822    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.742887 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.742874    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.742997 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.742968    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.744844 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.744825    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.745011 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.744992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd0aaa95-f669-4e19-87bf-5d97d9532618-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.752096 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.752072    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq6qf\" (UniqueName: \"kubernetes.io/projected/fd0aaa95-f669-4e19-87bf-5d97d9532618-kube-api-access-qq6qf\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:03.943327 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:03.943205    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:04.092975 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:04.092941    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"]
Jul 15 11:10:04.096713 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:10:04.096679    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0aaa95_f669_4e19_87bf_5d97d9532618.slice/crio-14afc83d4e145deefbc0f79b98ea4ecd9cf15c52ed440c001d6e91a4321b8df4 WatchSource:0}: Error finding container 14afc83d4e145deefbc0f79b98ea4ecd9cf15c52ed440c001d6e91a4321b8df4: Status 404 returned error can't find the container with id 14afc83d4e145deefbc0f79b98ea4ecd9cf15c52ed440c001d6e91a4321b8df4
Jul 15 11:10:04.669974 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:04.669935    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" event={"ID":"fd0aaa95-f669-4e19-87bf-5d97d9532618","Type":"ContainerStarted","Data":"84a40821ff32f311761ba95dff2693f442e94f77f09e733419c313dbc6009c7f"}
Jul 15 11:10:04.670457 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:04.669997    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" event={"ID":"fd0aaa95-f669-4e19-87bf-5d97d9532618","Type":"ContainerStarted","Data":"14afc83d4e145deefbc0f79b98ea4ecd9cf15c52ed440c001d6e91a4321b8df4"}
Jul 15 11:10:07.686697 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:07.686662    2566 generic.go:356] "Generic (PLEG): container finished" podID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerID="84a40821ff32f311761ba95dff2693f442e94f77f09e733419c313dbc6009c7f" exitCode=0
Jul 15 11:10:07.687081 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:07.686718    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" event={"ID":"fd0aaa95-f669-4e19-87bf-5d97d9532618","Type":"ContainerDied","Data":"84a40821ff32f311761ba95dff2693f442e94f77f09e733419c313dbc6009c7f"}
Jul 15 11:10:08.647003 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.646962    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"]
Jul 15 11:10:08.650488 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.650468    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.653692 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.653669    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 15 11:10:08.654656 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.654639    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 15 11:10:08.667524 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.667463    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"]
Jul 15 11:10:08.677785 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.677755    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.677929 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.677795    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c89e3c5b-09e9-43cd-a100-4960c89c1c84-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.677929 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.677839    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.677929 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.677861    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.677929 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.677893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.678095 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.677930    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.678095 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.677991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hkfq\" (UniqueName: \"kubernetes.io/projected/c89e3c5b-09e9-43cd-a100-4960c89c1c84-kube-api-access-9hkfq\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.678095 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.678018    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89e3c5b-09e9-43cd-a100-4960c89c1c84-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.693791 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.693759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" event={"ID":"fd0aaa95-f669-4e19-87bf-5d97d9532618","Type":"ContainerStarted","Data":"41eae1101ceaddd62737c54abe1b7d4df811d179bf4664ab88e90cba79c6f012"}
Jul 15 11:10:08.716855 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.716800    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" podStartSLOduration=2.736161251 podStartE2EDuration="5.716786313s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.980625062s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:10:08.715626191 +0000 UTC m=+1477.945593723" watchObservedRunningTime="2026-07-15 11:10:08.716786313 +0000 UTC m=+1477.946753844"
Jul 15 11:10:08.778559 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.778490    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.778559 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.778552    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.778825 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.778592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.778825 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.778610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.778825 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.778634    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hkfq\" (UniqueName: \"kubernetes.io/projected/c89e3c5b-09e9-43cd-a100-4960c89c1c84-kube-api-access-9hkfq\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.778825 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.778656    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89e3c5b-09e9-43cd-a100-4960c89c1c84-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.778825 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.778738    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.778825 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.778766    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c89e3c5b-09e9-43cd-a100-4960c89c1c84-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.779176 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.778932    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.779176 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.779004    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.779366 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.779328    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.779532 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.779489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.780020 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.779998    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c89e3c5b-09e9-43cd-a100-4960c89c1c84-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.780964 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.780943    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.781359 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.781339    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89e3c5b-09e9-43cd-a100-4960c89c1c84-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.788578 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.788547    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hkfq\" (UniqueName: \"kubernetes.io/projected/c89e3c5b-09e9-43cd-a100-4960c89c1c84-kube-api-access-9hkfq\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:08.943988 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.943892    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:08.961205 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:08.961163    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:09.116964 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:09.116929    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"]
Jul 15 11:10:09.118296 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:10:09.118271    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89e3c5b_09e9_43cd_a100_4960c89c1c84.slice/crio-3fe1aa4d635e4f3baf174252ea8999ab59536c637ac297b684a677987dd184d2 WatchSource:0}: Error finding container 3fe1aa4d635e4f3baf174252ea8999ab59536c637ac297b684a677987dd184d2: Status 404 returned error can't find the container with id 3fe1aa4d635e4f3baf174252ea8999ab59536c637ac297b684a677987dd184d2
Jul 15 11:10:09.700533 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:09.700470    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt" event={"ID":"c89e3c5b-09e9-43cd-a100-4960c89c1c84","Type":"ContainerStarted","Data":"b2df56b6e13e9e9017bb42cd137f9ca9e55d346af3d1d890dcce358753451131"}
Jul 15 11:10:09.700980 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:09.700559    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt" event={"ID":"c89e3c5b-09e9-43cd-a100-4960c89c1c84","Type":"ContainerStarted","Data":"3fe1aa4d635e4f3baf174252ea8999ab59536c637ac297b684a677987dd184d2"}
Jul 15 11:10:12.718948 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:12.718860    2566 generic.go:356] "Generic (PLEG): container finished" podID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" containerID="b2df56b6e13e9e9017bb42cd137f9ca9e55d346af3d1d890dcce358753451131" exitCode=0
Jul 15 11:10:12.719327 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:12.718942    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt" event={"ID":"c89e3c5b-09e9-43cd-a100-4960c89c1c84","Type":"ContainerDied","Data":"b2df56b6e13e9e9017bb42cd137f9ca9e55d346af3d1d890dcce358753451131"}
Jul 15 11:10:13.725898 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:13.725863    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt" event={"ID":"c89e3c5b-09e9-43cd-a100-4960c89c1c84","Type":"ContainerStarted","Data":"b4e7587019e92b2179330017bcd6e0ccdeb2d1e5e1d03a966121a69ac2b691e2"}
Jul 15 11:10:13.759434 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:13.759379    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt" podStartSLOduration=2.535386549 podStartE2EDuration="5.759363145s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.223976596s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:10:13.75568432 +0000 UTC m=+1482.985651851" watchObservedRunningTime="2026-07-15 11:10:13.759363145 +0000 UTC m=+1482.989330677"
Jul 15 11:10:13.944213 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:13.944116    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:10:13.945985 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:13.945948    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 15 11:10:13.961857 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:13.961810    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:18.962145 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:18.962109    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:10:18.971891 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:18.967208    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt" podUID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 15 11:10:23.944808 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:23.944759    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 15 11:10:28.962553 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:28.962479    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt" podUID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 15 11:10:33.944777 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:33.944724    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 15 11:10:38.232024 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:38.231993    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:10:38.232868 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:38.232842    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:10:38.961893 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:38.961672    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt" podUID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 15 11:10:43.943974 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:43.943931    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 15 11:10:48.961894 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:48.961847    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt" podUID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 15 11:10:53.945541 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:53.945403    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 15 11:10:58.961743 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:10:58.961694    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt" podUID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 15 11:11:03.944517 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:11:03.944443    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 15 11:11:08.962239 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:11:08.962128    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt" podUID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 15 11:11:13.944663 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:11:13.944563    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 15 11:11:18.962482 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:11:18.962416    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt" podUID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 15 11:11:23.944801 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:11:23.944752    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 15 11:11:28.962321 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:11:28.962273    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt" podUID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 15 11:11:33.944433 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:11:33.944372    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 15 11:11:38.979553 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:11:38.979490    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:11:38.993187 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:11:38.993151    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:11:43.944634 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:11:43.944591    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 15 11:11:53.944463 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:11:53.944414    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 15 11:12:03.944649 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:03.944597    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 15 11:12:13.954336 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:13.954255    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:12:13.967297 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:13.967265    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:12:18.628133 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:18.628102    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"]
Jul 15 11:12:30.466645 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:30.466610    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:12:30.467143 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:30.466670    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:12:30.469441 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:30.469411    2566 generic.go:356] "Generic (PLEG): container finished" podID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerID="4eeb58903d01e6aa70228839108c9fd2b05daeb403c607bdd60ab020f92113c5" exitCode=137
Jul 15 11:12:30.469608 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:30.469477    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" event={"ID":"63aad8c8-2687-4adf-9bab-9c36b0086419","Type":"ContainerDied","Data":"4eeb58903d01e6aa70228839108c9fd2b05daeb403c607bdd60ab020f92113c5"}
Jul 15 11:12:30.469608 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:30.469579    2566 scope.go:117] "RemoveContainer" containerID="f27a0f4bd53ee0ab22a097017bf96977f419eb0c50ff78680f9b8c16d279111a"
Jul 15 11:12:30.469991 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:30.469967    2566 scope.go:117] "RemoveContainer" containerID="4eeb58903d01e6aa70228839108c9fd2b05daeb403c607bdd60ab020f92113c5"
Jul 15 11:12:30.470203 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:12:30.470181    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-866675cbb8-fmhhm_openshift-ingress(63aad8c8-2687-4adf-9bab-9c36b0086419)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419"
Jul 15 11:12:33.651924 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:33.651888    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="main" containerID="cri-o://41eae1101ceaddd62737c54abe1b7d4df811d179bf4664ab88e90cba79c6f012" gracePeriod=45
Jul 15 11:12:33.955066 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:33.954970    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 15 11:12:34.411371 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.411348    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:12:34.446088 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.446054    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd0aaa95-f669-4e19-87bf-5d97d9532618-tls-certs\") pod \"fd0aaa95-f669-4e19-87bf-5d97d9532618\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") "
Jul 15 11:12:34.446252 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.446109    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-tmp-dir\") pod \"fd0aaa95-f669-4e19-87bf-5d97d9532618\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") "
Jul 15 11:12:34.446252 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.446130    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-home\") pod \"fd0aaa95-f669-4e19-87bf-5d97d9532618\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") "
Jul 15 11:12:34.446252 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.446157    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq6qf\" (UniqueName: \"kubernetes.io/projected/fd0aaa95-f669-4e19-87bf-5d97d9532618-kube-api-access-qq6qf\") pod \"fd0aaa95-f669-4e19-87bf-5d97d9532618\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") "
Jul 15 11:12:34.446252 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.446187    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-kserve-provision-location\") pod \"fd0aaa95-f669-4e19-87bf-5d97d9532618\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") "
Jul 15 11:12:34.446252 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.446229    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-dshm\") pod \"fd0aaa95-f669-4e19-87bf-5d97d9532618\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") "
Jul 15 11:12:34.446555 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.446265    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-model-cache\") pod \"fd0aaa95-f669-4e19-87bf-5d97d9532618\" (UID: \"fd0aaa95-f669-4e19-87bf-5d97d9532618\") "
Jul 15 11:12:34.446720 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.446679    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd0aaa95-f669-4e19-87bf-5d97d9532618" (UID: "fd0aaa95-f669-4e19-87bf-5d97d9532618"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:12:34.447144 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.447119    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-home" (OuterVolumeSpecName: "home") pod "fd0aaa95-f669-4e19-87bf-5d97d9532618" (UID: "fd0aaa95-f669-4e19-87bf-5d97d9532618"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:12:34.448675 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.448642    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0aaa95-f669-4e19-87bf-5d97d9532618-kube-api-access-qq6qf" (OuterVolumeSpecName: "kube-api-access-qq6qf") pod "fd0aaa95-f669-4e19-87bf-5d97d9532618" (UID: "fd0aaa95-f669-4e19-87bf-5d97d9532618"). InnerVolumeSpecName "kube-api-access-qq6qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:12:34.448928 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.448897    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-dshm" (OuterVolumeSpecName: "dshm") pod "fd0aaa95-f669-4e19-87bf-5d97d9532618" (UID: "fd0aaa95-f669-4e19-87bf-5d97d9532618"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:12:34.449033 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.448983    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd0aaa95-f669-4e19-87bf-5d97d9532618-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd0aaa95-f669-4e19-87bf-5d97d9532618" (UID: "fd0aaa95-f669-4e19-87bf-5d97d9532618"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:12:34.460042 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.460005    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd0aaa95-f669-4e19-87bf-5d97d9532618" (UID: "fd0aaa95-f669-4e19-87bf-5d97d9532618"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:12:34.491590 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.491556    2566 generic.go:356] "Generic (PLEG): container finished" podID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerID="41eae1101ceaddd62737c54abe1b7d4df811d179bf4664ab88e90cba79c6f012" exitCode=0
Jul 15 11:12:34.491787 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.491612    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"
Jul 15 11:12:34.491787 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.491640    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" event={"ID":"fd0aaa95-f669-4e19-87bf-5d97d9532618","Type":"ContainerDied","Data":"41eae1101ceaddd62737c54abe1b7d4df811d179bf4664ab88e90cba79c6f012"}
Jul 15 11:12:34.491787 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.491699    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt" event={"ID":"fd0aaa95-f669-4e19-87bf-5d97d9532618","Type":"ContainerDied","Data":"14afc83d4e145deefbc0f79b98ea4ecd9cf15c52ed440c001d6e91a4321b8df4"}
Jul 15 11:12:34.491787 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.491726    2566 scope.go:117] "RemoveContainer" containerID="41eae1101ceaddd62737c54abe1b7d4df811d179bf4664ab88e90cba79c6f012"
Jul 15 11:12:34.500878 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.500856    2566 scope.go:117] "RemoveContainer" containerID="84a40821ff32f311761ba95dff2693f442e94f77f09e733419c313dbc6009c7f"
Jul 15 11:12:34.508822 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.508805    2566 scope.go:117] "RemoveContainer" containerID="41eae1101ceaddd62737c54abe1b7d4df811d179bf4664ab88e90cba79c6f012"
Jul 15 11:12:34.509084 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:12:34.509065    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41eae1101ceaddd62737c54abe1b7d4df811d179bf4664ab88e90cba79c6f012\": container with ID starting with 41eae1101ceaddd62737c54abe1b7d4df811d179bf4664ab88e90cba79c6f012 not found: ID does not exist" containerID="41eae1101ceaddd62737c54abe1b7d4df811d179bf4664ab88e90cba79c6f012"
Jul 15 11:12:34.509129 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.509093    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41eae1101ceaddd62737c54abe1b7d4df811d179bf4664ab88e90cba79c6f012"} err="failed to get container status \"41eae1101ceaddd62737c54abe1b7d4df811d179bf4664ab88e90cba79c6f012\": rpc error: code = NotFound desc = could not find container \"41eae1101ceaddd62737c54abe1b7d4df811d179bf4664ab88e90cba79c6f012\": container with ID starting with 41eae1101ceaddd62737c54abe1b7d4df811d179bf4664ab88e90cba79c6f012 not found: ID does not exist"
Jul 15 11:12:34.509129 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.509112    2566 scope.go:117] "RemoveContainer" containerID="84a40821ff32f311761ba95dff2693f442e94f77f09e733419c313dbc6009c7f"
Jul 15 11:12:34.509333 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:12:34.509314    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a40821ff32f311761ba95dff2693f442e94f77f09e733419c313dbc6009c7f\": container with ID starting with 84a40821ff32f311761ba95dff2693f442e94f77f09e733419c313dbc6009c7f not found: ID does not exist" containerID="84a40821ff32f311761ba95dff2693f442e94f77f09e733419c313dbc6009c7f"
Jul 15 11:12:34.509373 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.509337    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a40821ff32f311761ba95dff2693f442e94f77f09e733419c313dbc6009c7f"} err="failed to get container status \"84a40821ff32f311761ba95dff2693f442e94f77f09e733419c313dbc6009c7f\": rpc error: code = NotFound desc = could not find container \"84a40821ff32f311761ba95dff2693f442e94f77f09e733419c313dbc6009c7f\": container with ID starting with 84a40821ff32f311761ba95dff2693f442e94f77f09e733419c313dbc6009c7f not found: ID does not exist"
Jul 15 11:12:34.518885 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.518829    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fd0aaa95-f669-4e19-87bf-5d97d9532618" (UID: "fd0aaa95-f669-4e19-87bf-5d97d9532618"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:12:34.546936 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.546867    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd0aaa95-f669-4e19-87bf-5d97d9532618-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:12:34.546936 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.546911    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:12:34.546936 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.546920    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:12:34.546936 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.546929    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq6qf\" (UniqueName: \"kubernetes.io/projected/fd0aaa95-f669-4e19-87bf-5d97d9532618-kube-api-access-qq6qf\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:12:34.546936 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.546939    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:12:34.547169 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.546948    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:12:34.547169 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.546956    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd0aaa95-f669-4e19-87bf-5d97d9532618-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:12:34.814762 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.814736    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"]
Jul 15 11:12:34.823306 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:34.823283    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7976c6bff9wsfwt"]
Jul 15 11:12:35.268104 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:35.268073    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" path="/var/lib/kubelet/pods/fd0aaa95-f669-4e19-87bf-5d97d9532618/volumes"
Jul 15 11:12:42.423216 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.423183    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"]
Jul 15 11:12:42.423650 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.423567    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="main"
Jul 15 11:12:42.423650 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.423580    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="main"
Jul 15 11:12:42.423650 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.423590    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="storage-initializer"
Jul 15 11:12:42.423650 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.423596    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="storage-initializer"
Jul 15 11:12:42.423650 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.423650    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd0aaa95-f669-4e19-87bf-5d97d9532618" containerName="main"
Jul 15 11:12:42.428273 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.428253    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.430884 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.430866    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 15 11:12:42.444569 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.444547    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"]
Jul 15 11:12:42.519068 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.519033    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.519242 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.519078    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-dshm\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.519242 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.519113    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-home\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.519242 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.519128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-model-cache\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.519242 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.519149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-tls-certs\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.519242 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.519173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r956f\" (UniqueName: \"kubernetes.io/projected/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-kube-api-access-r956f\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.519242 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.519230    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-tmp-dir\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.620462 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.620428    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-home\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.620462 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.620461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-model-cache\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.620686 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.620482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-tls-certs\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.620686 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.620532    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r956f\" (UniqueName: \"kubernetes.io/projected/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-kube-api-access-r956f\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.620686 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.620555    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-tmp-dir\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.620686 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.620601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.620686 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.620647    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-dshm\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.620978 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.620900    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-model-cache\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.621038 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.620978    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-home\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.621038 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.621004    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-tmp-dir\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.621116 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.621034    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.622946 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.622926    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-dshm\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.623111 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.623092    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-tls-certs\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.632403 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.632367    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r956f\" (UniqueName: \"kubernetes.io/projected/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-kube-api-access-r956f\") pod \"custom-route-timeout-test-kserve-6964567f65-7xrv5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.737755 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.737654    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:42.886262 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:42.886230    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"]
Jul 15 11:12:42.889560 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:12:42.889535    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658bbe89_64f9_4afd_abb7_e8e6aa7c91d5.slice/crio-957c3951c0cf6f008817a427a1c441925727cd011b910afb15b0e27294a99f46 WatchSource:0}: Error finding container 957c3951c0cf6f008817a427a1c441925727cd011b910afb15b0e27294a99f46: Status 404 returned error can't find the container with id 957c3951c0cf6f008817a427a1c441925727cd011b910afb15b0e27294a99f46
Jul 15 11:12:43.533230 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:43.533190    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" event={"ID":"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5","Type":"ContainerStarted","Data":"e375babd99b015c63499302d483a0d4c792c6811a94fec14b5f9f1a859d0e3f2"}
Jul 15 11:12:43.533650 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:43.533260    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" event={"ID":"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5","Type":"ContainerStarted","Data":"957c3951c0cf6f008817a427a1c441925727cd011b910afb15b0e27294a99f46"}
Jul 15 11:12:45.262473 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:45.262433    2566 scope.go:117] "RemoveContainer" containerID="4eeb58903d01e6aa70228839108c9fd2b05daeb403c607bdd60ab020f92113c5"
Jul 15 11:12:45.262898 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:12:45.262624    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-866675cbb8-fmhhm_openshift-ingress(63aad8c8-2687-4adf-9bab-9c36b0086419)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419"
Jul 15 11:12:45.465831 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:45.465788    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:12:45.542894 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:45.542816    2566 scope.go:117] "RemoveContainer" containerID="4eeb58903d01e6aa70228839108c9fd2b05daeb403c607bdd60ab020f92113c5"
Jul 15 11:12:45.543045 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:12:45.543004    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-866675cbb8-fmhhm_openshift-ingress(63aad8c8-2687-4adf-9bab-9c36b0086419)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419"
Jul 15 11:12:46.547736 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:46.547704    2566 generic.go:356] "Generic (PLEG): container finished" podID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerID="e375babd99b015c63499302d483a0d4c792c6811a94fec14b5f9f1a859d0e3f2" exitCode=0
Jul 15 11:12:46.548149 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:46.547778    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" event={"ID":"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5","Type":"ContainerDied","Data":"e375babd99b015c63499302d483a0d4c792c6811a94fec14b5f9f1a859d0e3f2"}
Jul 15 11:12:47.553863 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:47.553829    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" event={"ID":"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5","Type":"ContainerStarted","Data":"95678d2b14719dc7e36e966edb41134390c1adb93590a8372fea1ed0051b0850"}
Jul 15 11:12:47.579662 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:47.579615    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" podStartSLOduration=2.641451253 podStartE2EDuration="5.579600811s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.938149558s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:12:47.577151656 +0000 UTC m=+1636.807119187" watchObservedRunningTime="2026-07-15 11:12:47.579600811 +0000 UTC m=+1636.809568353"
Jul 15 11:12:47.738644 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:47.738611    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:52.737766 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:52.737733    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:12:52.739407 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:52.739375    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Jul 15 11:12:59.262106 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:59.262078    2566 scope.go:117] "RemoveContainer" containerID="4eeb58903d01e6aa70228839108c9fd2b05daeb403c607bdd60ab020f92113c5"
Jul 15 11:12:59.263334 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:59.263300    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:12:59.263442 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:59.263391    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:12:59.263525 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:59.263444    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:12:59.611563 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:59.611456    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" event={"ID":"63aad8c8-2687-4adf-9bab-9c36b0086419","Type":"ContainerStarted","Data":"c702d4dc1d5d36e2fc85a740db118bcc241cf44e5182003066b91c2ac40c25b7"}
Jul 15 11:12:59.797331 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:12:59.797302    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:13:00.465875 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:00.465837    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:13:00.798558 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:00.798520    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:13:00.798744 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:00.798601    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:13:01.798302 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:01.798258    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:13:01.798709 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:01.798326    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:13:02.738431 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:02.738390    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Jul 15 11:13:02.798382 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:02.798333    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:13:02.798808 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:02.798394    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:13:03.798261 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:03.798221    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:13:03.798449 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:03.798279    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:13:04.797766 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:04.797733    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:13:04.797979 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:04.797785    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:13:06.701019 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:06.700987    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:13:06.702384 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:06.702359    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:13:12.738469 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:12.738423    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Jul 15 11:13:22.738420 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:22.738366    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Jul 15 11:13:32.738366 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:32.738302    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Jul 15 11:13:42.738343 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:42.738300    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Jul 15 11:13:44.845418 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:44.845323    2566 generic.go:356] "Generic (PLEG): container finished" podID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerID="c702d4dc1d5d36e2fc85a740db118bcc241cf44e5182003066b91c2ac40c25b7" exitCode=137
Jul 15 11:13:44.845418 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:44.845390    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" event={"ID":"63aad8c8-2687-4adf-9bab-9c36b0086419","Type":"ContainerDied","Data":"c702d4dc1d5d36e2fc85a740db118bcc241cf44e5182003066b91c2ac40c25b7"}
Jul 15 11:13:44.845919 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:44.845453    2566 scope.go:117] "RemoveContainer" containerID="4eeb58903d01e6aa70228839108c9fd2b05daeb403c607bdd60ab020f92113c5"
Jul 15 11:13:44.845963 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:44.845928    2566 scope.go:117] "RemoveContainer" containerID="c702d4dc1d5d36e2fc85a740db118bcc241cf44e5182003066b91c2ac40c25b7"
Jul 15 11:13:44.846168 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:13:44.846146    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-866675cbb8-fmhhm_openshift-ingress(63aad8c8-2687-4adf-9bab-9c36b0086419)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419"
Jul 15 11:13:51.702238 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:51.702152    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:13:51.702789 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:51.702609    2566 scope.go:117] "RemoveContainer" containerID="c702d4dc1d5d36e2fc85a740db118bcc241cf44e5182003066b91c2ac40c25b7"
Jul 15 11:13:51.702868 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:13:51.702826    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-866675cbb8-fmhhm_openshift-ingress(63aad8c8-2687-4adf-9bab-9c36b0086419)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419"
Jul 15 11:13:52.738043 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:13:52.738002    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Jul 15 11:14:02.738515 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:02.738456    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Jul 15 11:14:03.262073 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:03.262038    2566 scope.go:117] "RemoveContainer" containerID="c702d4dc1d5d36e2fc85a740db118bcc241cf44e5182003066b91c2ac40c25b7"
Jul 15 11:14:03.262292 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:14:03.262240    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-866675cbb8-fmhhm_openshift-ingress(63aad8c8-2687-4adf-9bab-9c36b0086419)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419"
Jul 15 11:14:12.738809 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:12.738770    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Jul 15 11:14:14.262554 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:14.262511    2566 scope.go:117] "RemoveContainer" containerID="c702d4dc1d5d36e2fc85a740db118bcc241cf44e5182003066b91c2ac40c25b7"
Jul 15 11:14:14.263022 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:14:14.262919    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 40s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-866675cbb8-fmhhm_openshift-ingress(63aad8c8-2687-4adf-9bab-9c36b0086419)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419"
Jul 15 11:14:22.738443 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:22.738335    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Jul 15 11:14:29.262352 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:29.262307    2566 scope.go:117] "RemoveContainer" containerID="c702d4dc1d5d36e2fc85a740db118bcc241cf44e5182003066b91c2ac40c25b7"
Jul 15 11:14:29.263636 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:29.263604    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:14:29.263764 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:29.263678    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:14:29.263764 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:29.263710    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:14:30.066624 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:30.066556    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" event={"ID":"63aad8c8-2687-4adf-9bab-9c36b0086419","Type":"ContainerStarted","Data":"629e590ed2e52381978bdec15963f9c85350f3b108280dcf4a88baac37f1e057"}
Jul 15 11:14:30.797616 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:30.797576    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:14:30.798350 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:30.798319    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:14:30.798448 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:30.798396    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:14:31.797945 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:31.797900    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:14:31.798352 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:31.797987    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:14:32.738619 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:32.738567    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Jul 15 11:14:32.798338 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:32.798299    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:14:32.798774 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:32.798369    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:14:33.797792 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:33.797749    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:14:33.797973 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:33.797841    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:14:34.798420 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:34.798381    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:14:34.798827 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:34.798453    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:14:36.617002 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:36.616970    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:14:36.617530 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:36.617402    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:14:36.618399 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:36.618367    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:14:42.749368 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:42.749341    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:14:42.757474 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:42.757448    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:14:48.889943 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:14:48.889901    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"]
Jul 15 11:15:03.913058 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:03.912979    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerName="main" containerID="cri-o://95678d2b14719dc7e36e966edb41134390c1adb93590a8372fea1ed0051b0850" gracePeriod=45
Jul 15 11:15:04.684009 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.683983    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:15:04.811149 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.811112    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-kserve-provision-location\") pod \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") "
Jul 15 11:15:04.811149 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.811151    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-tmp-dir\") pod \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") "
Jul 15 11:15:04.811415 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.811206    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r956f\" (UniqueName: \"kubernetes.io/projected/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-kube-api-access-r956f\") pod \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") "
Jul 15 11:15:04.811415 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.811224    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-home\") pod \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") "
Jul 15 11:15:04.811415 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.811261    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-dshm\") pod \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") "
Jul 15 11:15:04.811415 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.811334    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-tls-certs\") pod \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") "
Jul 15 11:15:04.811415 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.811368    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-model-cache\") pod \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\" (UID: \"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5\") "
Jul 15 11:15:04.811832 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.811801    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" (UID: "658bbe89-64f9-4afd-abb7-e8e6aa7c91d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:15:04.812021 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.811974    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-home" (OuterVolumeSpecName: "home") pod "658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" (UID: "658bbe89-64f9-4afd-abb7-e8e6aa7c91d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:15:04.813914 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.813888    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-kube-api-access-r956f" (OuterVolumeSpecName: "kube-api-access-r956f") pod "658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" (UID: "658bbe89-64f9-4afd-abb7-e8e6aa7c91d5"). InnerVolumeSpecName "kube-api-access-r956f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:15:04.814048 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.814021    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-dshm" (OuterVolumeSpecName: "dshm") pod "658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" (UID: "658bbe89-64f9-4afd-abb7-e8e6aa7c91d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:15:04.814215 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.814195    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" (UID: "658bbe89-64f9-4afd-abb7-e8e6aa7c91d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:15:04.826086 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.826061    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" (UID: "658bbe89-64f9-4afd-abb7-e8e6aa7c91d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:15:04.880919 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.880874    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" (UID: "658bbe89-64f9-4afd-abb7-e8e6aa7c91d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:15:04.912260 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.912227    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:15:04.912260 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.912256    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:15:04.912260 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.912265    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:15:04.912513 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.912273    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:15:04.912513 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.912282    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r956f\" (UniqueName: \"kubernetes.io/projected/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-kube-api-access-r956f\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:15:04.912513 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.912290    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:15:04.912513 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:04.912300    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:15:05.248133 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:05.248100    2566 generic.go:356] "Generic (PLEG): container finished" podID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerID="95678d2b14719dc7e36e966edb41134390c1adb93590a8372fea1ed0051b0850" exitCode=0
Jul 15 11:15:05.248532 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:05.248149    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"
Jul 15 11:15:05.248532 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:05.248180    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" event={"ID":"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5","Type":"ContainerDied","Data":"95678d2b14719dc7e36e966edb41134390c1adb93590a8372fea1ed0051b0850"}
Jul 15 11:15:05.248532 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:05.248242    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5" event={"ID":"658bbe89-64f9-4afd-abb7-e8e6aa7c91d5","Type":"ContainerDied","Data":"957c3951c0cf6f008817a427a1c441925727cd011b910afb15b0e27294a99f46"}
Jul 15 11:15:05.248532 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:05.248278    2566 scope.go:117] "RemoveContainer" containerID="95678d2b14719dc7e36e966edb41134390c1adb93590a8372fea1ed0051b0850"
Jul 15 11:15:05.257983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:05.257957    2566 scope.go:117] "RemoveContainer" containerID="e375babd99b015c63499302d483a0d4c792c6811a94fec14b5f9f1a859d0e3f2"
Jul 15 11:15:05.266875 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:05.266853    2566 scope.go:117] "RemoveContainer" containerID="95678d2b14719dc7e36e966edb41134390c1adb93590a8372fea1ed0051b0850"
Jul 15 11:15:05.267151 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:15:05.267124    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95678d2b14719dc7e36e966edb41134390c1adb93590a8372fea1ed0051b0850\": container with ID starting with 95678d2b14719dc7e36e966edb41134390c1adb93590a8372fea1ed0051b0850 not found: ID does not exist" containerID="95678d2b14719dc7e36e966edb41134390c1adb93590a8372fea1ed0051b0850"
Jul 15 11:15:05.267279 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:05.267158    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95678d2b14719dc7e36e966edb41134390c1adb93590a8372fea1ed0051b0850"} err="failed to get container status \"95678d2b14719dc7e36e966edb41134390c1adb93590a8372fea1ed0051b0850\": rpc error: code = NotFound desc = could not find container \"95678d2b14719dc7e36e966edb41134390c1adb93590a8372fea1ed0051b0850\": container with ID starting with 95678d2b14719dc7e36e966edb41134390c1adb93590a8372fea1ed0051b0850 not found: ID does not exist"
Jul 15 11:15:05.267279 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:05.267180    2566 scope.go:117] "RemoveContainer" containerID="e375babd99b015c63499302d483a0d4c792c6811a94fec14b5f9f1a859d0e3f2"
Jul 15 11:15:05.267434 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:15:05.267414    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e375babd99b015c63499302d483a0d4c792c6811a94fec14b5f9f1a859d0e3f2\": container with ID starting with e375babd99b015c63499302d483a0d4c792c6811a94fec14b5f9f1a859d0e3f2 not found: ID does not exist" containerID="e375babd99b015c63499302d483a0d4c792c6811a94fec14b5f9f1a859d0e3f2"
Jul 15 11:15:05.267513 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:05.267442    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e375babd99b015c63499302d483a0d4c792c6811a94fec14b5f9f1a859d0e3f2"} err="failed to get container status \"e375babd99b015c63499302d483a0d4c792c6811a94fec14b5f9f1a859d0e3f2\": rpc error: code = NotFound desc = could not find container \"e375babd99b015c63499302d483a0d4c792c6811a94fec14b5f9f1a859d0e3f2\": container with ID starting with e375babd99b015c63499302d483a0d4c792c6811a94fec14b5f9f1a859d0e3f2 not found: ID does not exist"
Jul 15 11:15:05.271777 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:05.271749    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"]
Jul 15 11:15:05.277309 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:05.277282    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6964567f65-7xrv5"]
Jul 15 11:15:07.266895 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:07.266861    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" path="/var/lib/kubelet/pods/658bbe89-64f9-4afd-abb7-e8e6aa7c91d5/volumes"
Jul 15 11:15:14.297731 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:14.297701    2566 generic.go:356] "Generic (PLEG): container finished" podID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerID="629e590ed2e52381978bdec15963f9c85350f3b108280dcf4a88baac37f1e057" exitCode=137
Jul 15 11:15:14.298233 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:14.297776    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" event={"ID":"63aad8c8-2687-4adf-9bab-9c36b0086419","Type":"ContainerDied","Data":"629e590ed2e52381978bdec15963f9c85350f3b108280dcf4a88baac37f1e057"}
Jul 15 11:15:14.298233 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:14.297834    2566 scope.go:117] "RemoveContainer" containerID="c702d4dc1d5d36e2fc85a740db118bcc241cf44e5182003066b91c2ac40c25b7"
Jul 15 11:15:14.298233 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:14.298225    2566 scope.go:117] "RemoveContainer" containerID="629e590ed2e52381978bdec15963f9c85350f3b108280dcf4a88baac37f1e057"
Jul 15 11:15:14.298412 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:15:14.298394    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-866675cbb8-fmhhm_openshift-ingress(63aad8c8-2687-4adf-9bab-9c36b0086419)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419"
Jul 15 11:15:16.241621 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.241589    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"]
Jul 15 11:15:16.242097 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.242076    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerName="storage-initializer"
Jul 15 11:15:16.242171 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.242100    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerName="storage-initializer"
Jul 15 11:15:16.242171 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.242138    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerName="main"
Jul 15 11:15:16.242171 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.242150    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerName="main"
Jul 15 11:15:16.242323 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.242227    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="658bbe89-64f9-4afd-abb7-e8e6aa7c91d5" containerName="main"
Jul 15 11:15:16.246310 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.246289    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.248943 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.248925    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 15 11:15:16.256285 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.256262    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"]
Jul 15 11:15:16.415703 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.415667    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-tls-certs\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.415888 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.415723    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-tmp-dir\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.415888 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.415787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-kserve-provision-location\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.415888 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.415854    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-dshm\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.415992 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.415894    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-home\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.415992 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.415917    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-model-cache\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.415992 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.415968    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsvpn\" (UniqueName: \"kubernetes.io/projected/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-kube-api-access-fsvpn\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.517364 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.517330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-kserve-provision-location\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.517364 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.517373    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-dshm\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.517620 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.517393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-home\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.517620 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.517420    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-model-cache\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.517620 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.517451    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsvpn\" (UniqueName: \"kubernetes.io/projected/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-kube-api-access-fsvpn\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.517620 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.517487    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-tls-certs\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.517620 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.517542    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-tmp-dir\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.517879 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.517825    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-kserve-provision-location\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.517879 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.517866    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-model-cache\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.517986 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.517912    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-tmp-dir\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.518046 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.518029    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-home\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.519897 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.519874    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-dshm\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.520106 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.520085    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-tls-certs\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.525908 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.525888    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsvpn\" (UniqueName: \"kubernetes.io/projected/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-kube-api-access-fsvpn\") pod \"router-with-refs-test-kserve-c9875db89-4wq4r\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.556343 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.556322    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:16.744726 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:16.744701    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"]
Jul 15 11:15:16.746452 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:15:16.746424    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2225d2e8_6c95_4ee1_8f6c_ac61e41008ef.slice/crio-9ece671a3f103ceab62fbde2bcb84aa2a82474a34501fa468ec53ef9069a93d7 WatchSource:0}: Error finding container 9ece671a3f103ceab62fbde2bcb84aa2a82474a34501fa468ec53ef9069a93d7: Status 404 returned error can't find the container with id 9ece671a3f103ceab62fbde2bcb84aa2a82474a34501fa468ec53ef9069a93d7
Jul 15 11:15:17.315159 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:17.315120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" event={"ID":"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef","Type":"ContainerStarted","Data":"69389544dfc0d7f10ce38cd72ae3432e1d611ef9bc9a92d6ee242951f01696de"}
Jul 15 11:15:17.315553 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:17.315175    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" event={"ID":"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef","Type":"ContainerStarted","Data":"9ece671a3f103ceab62fbde2bcb84aa2a82474a34501fa468ec53ef9069a93d7"}
Jul 15 11:15:20.330799 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:20.330766    2566 generic.go:356] "Generic (PLEG): container finished" podID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerID="69389544dfc0d7f10ce38cd72ae3432e1d611ef9bc9a92d6ee242951f01696de" exitCode=0
Jul 15 11:15:20.331202 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:20.330836    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" event={"ID":"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef","Type":"ContainerDied","Data":"69389544dfc0d7f10ce38cd72ae3432e1d611ef9bc9a92d6ee242951f01696de"}
Jul 15 11:15:21.338779 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:21.338738    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" event={"ID":"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef","Type":"ContainerStarted","Data":"4ddc087224c8873154073c3f9aa8922f475d2e77e4a55b5a4070df401bdd7f1d"}
Jul 15 11:15:21.364783 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:21.364732    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" podStartSLOduration=2.418957024 podStartE2EDuration="5.364717825s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.945760801s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:15:21.363042334 +0000 UTC m=+1790.593009864" watchObservedRunningTime="2026-07-15 11:15:21.364717825 +0000 UTC m=+1790.594685392"
Jul 15 11:15:21.557346 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:21.557278    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:21.617340 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:21.617253    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:15:21.617759 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:21.617730    2566 scope.go:117] "RemoveContainer" containerID="629e590ed2e52381978bdec15963f9c85350f3b108280dcf4a88baac37f1e057"
Jul 15 11:15:21.617955 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:15:21.617930    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-866675cbb8-fmhhm_openshift-ingress(63aad8c8-2687-4adf-9bab-9c36b0086419)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419"
Jul 15 11:15:26.556789 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:26.556752    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:15:26.558444 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:26.558420    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 15 11:15:34.262249 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:34.262223    2566 scope.go:117] "RemoveContainer" containerID="629e590ed2e52381978bdec15963f9c85350f3b108280dcf4a88baac37f1e057"
Jul 15 11:15:34.262730 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:15:34.262418    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-866675cbb8-fmhhm_openshift-ingress(63aad8c8-2687-4adf-9bab-9c36b0086419)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419"
Jul 15 11:15:36.557297 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:36.557255    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 15 11:15:38.282970 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:38.282936    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:15:38.284997 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:38.284967    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:15:46.557522 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:46.557457    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 15 11:15:48.262240 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:48.262205    2566 scope.go:117] "RemoveContainer" containerID="629e590ed2e52381978bdec15963f9c85350f3b108280dcf4a88baac37f1e057"
Jul 15 11:15:48.262657 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:15:48.262392    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-866675cbb8-fmhhm_openshift-ingress(63aad8c8-2687-4adf-9bab-9c36b0086419)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419"
Jul 15 11:15:56.556715 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:15:56.556678    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 15 11:16:02.262859 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:02.262829    2566 scope.go:117] "RemoveContainer" containerID="629e590ed2e52381978bdec15963f9c85350f3b108280dcf4a88baac37f1e057"
Jul 15 11:16:02.263254 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:16:02.263012    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-866675cbb8-fmhhm_openshift-ingress(63aad8c8-2687-4adf-9bab-9c36b0086419)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419"
Jul 15 11:16:06.556963 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:06.556914    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 15 11:16:16.557596 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:16.557546    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 15 11:16:17.262443 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:17.262412    2566 scope.go:117] "RemoveContainer" containerID="629e590ed2e52381978bdec15963f9c85350f3b108280dcf4a88baac37f1e057"
Jul 15 11:16:17.262704 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:16:17.262678    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-866675cbb8-fmhhm_openshift-ingress(63aad8c8-2687-4adf-9bab-9c36b0086419)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419"
Jul 15 11:16:26.557849 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:26.557796    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 15 11:16:31.264180 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:31.264149    2566 scope.go:117] "RemoveContainer" containerID="629e590ed2e52381978bdec15963f9c85350f3b108280dcf4a88baac37f1e057"
Jul 15 11:16:31.264593 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:16:31.264338    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-866675cbb8-fmhhm_openshift-ingress(63aad8c8-2687-4adf-9bab-9c36b0086419)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419"
Jul 15 11:16:36.556880 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:36.556830    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 15 11:16:45.262843 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:45.262815    2566 scope.go:117] "RemoveContainer" containerID="629e590ed2e52381978bdec15963f9c85350f3b108280dcf4a88baac37f1e057"
Jul 15 11:16:45.264136 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:45.264096    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:16:45.264215 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:45.264195    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:16:45.264278 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:45.264260    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:16:45.733041 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:45.732949    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" event={"ID":"63aad8c8-2687-4adf-9bab-9c36b0086419","Type":"ContainerStarted","Data":"83521586f29daec561b6d2b25ad55b714de9c32302038bdf0cdfa08b59cca0a3"}
Jul 15 11:16:45.797442 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:45.797409    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:16:46.557725 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:46.557677    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 15 11:16:46.798769 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:46.798732    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:16:46.798965 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:46.798794    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:16:47.798458 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:47.798416    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:16:47.798881 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:47.798478    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:16:48.797859 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:48.797823    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:16:48.798049 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:48.797877    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:16:49.797885 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:49.797843    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:16:49.798283 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:49.797913    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:16:50.798086 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:50.798045    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:16:50.798480 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:50.798112    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:16:51.617884 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:51.617804    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:16:51.797727 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:51.797690    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:16:51.797915 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:51.797764    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:16:53.423599 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:53.423553    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:16:53.425588 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:53.425563    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:16:56.557869 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:16:56.557806    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 15 11:17:06.557751 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:06.557694    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 15 11:17:16.568240 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:16.568200    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:17:16.576522 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:16.576483    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:17:21.992086 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:21.992056    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"]
Jul 15 11:17:37.016254 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.016195    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="main" containerID="cri-o://4ddc087224c8873154073c3f9aa8922f475d2e77e4a55b5a4070df401bdd7f1d" gracePeriod=45
Jul 15 11:17:37.569531 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.569479    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Jul 15 11:17:37.789611 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.789585    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:17:37.876906 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.876805    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsvpn\" (UniqueName: \"kubernetes.io/projected/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-kube-api-access-fsvpn\") pod \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") "
Jul 15 11:17:37.876906 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.876868    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-kserve-provision-location\") pod \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") "
Jul 15 11:17:37.876906 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.876904    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-home\") pod \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") "
Jul 15 11:17:37.877210 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.876938    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-dshm\") pod \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") "
Jul 15 11:17:37.877312 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.877017    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-model-cache\") pod \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") "
Jul 15 11:17:37.877476 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.877461    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-tls-certs\") pod \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") "
Jul 15 11:17:37.877627 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.877612    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-tmp-dir\") pod \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\" (UID: \"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef\") "
Jul 15 11:17:37.877879 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.877628    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" (UID: "2225d2e8-6c95-4ee1-8f6c-ac61e41008ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:17:37.877879 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.877719    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-home" (OuterVolumeSpecName: "home") pod "2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" (UID: "2225d2e8-6c95-4ee1-8f6c-ac61e41008ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:17:37.878291 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.878271    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:17:37.878400 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.878386    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:17:37.879921 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.879882    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-dshm" (OuterVolumeSpecName: "dshm") pod "2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" (UID: "2225d2e8-6c95-4ee1-8f6c-ac61e41008ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:17:37.897300 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.897269    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" (UID: "2225d2e8-6c95-4ee1-8f6c-ac61e41008ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:17:37.897399 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.897289    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-kube-api-access-fsvpn" (OuterVolumeSpecName: "kube-api-access-fsvpn") pod "2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" (UID: "2225d2e8-6c95-4ee1-8f6c-ac61e41008ef"). InnerVolumeSpecName "kube-api-access-fsvpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:17:37.897772 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.897747    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" (UID: "2225d2e8-6c95-4ee1-8f6c-ac61e41008ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:17:37.952758 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.952701    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" (UID: "2225d2e8-6c95-4ee1-8f6c-ac61e41008ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:17:37.978549 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.978513    2566 generic.go:356] "Generic (PLEG): container finished" podID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerID="4ddc087224c8873154073c3f9aa8922f475d2e77e4a55b5a4070df401bdd7f1d" exitCode=0
Jul 15 11:17:37.978744 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.978563    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" event={"ID":"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef","Type":"ContainerDied","Data":"4ddc087224c8873154073c3f9aa8922f475d2e77e4a55b5a4070df401bdd7f1d"}
Jul 15 11:17:37.978744 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.978596    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"
Jul 15 11:17:37.978744 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.978640    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r" event={"ID":"2225d2e8-6c95-4ee1-8f6c-ac61e41008ef","Type":"ContainerDied","Data":"9ece671a3f103ceab62fbde2bcb84aa2a82474a34501fa468ec53ef9069a93d7"}
Jul 15 11:17:37.978744 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.978669    2566 scope.go:117] "RemoveContainer" containerID="4ddc087224c8873154073c3f9aa8922f475d2e77e4a55b5a4070df401bdd7f1d"
Jul 15 11:17:37.978989 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.978962    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:17:37.979056 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.978994    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:17:37.979056 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.979010    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsvpn\" (UniqueName: \"kubernetes.io/projected/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-kube-api-access-fsvpn\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:17:37.979056 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.979023    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:17:37.979056 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.979037    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:17:37.987582 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.987563    2566 scope.go:117] "RemoveContainer" containerID="69389544dfc0d7f10ce38cd72ae3432e1d611ef9bc9a92d6ee242951f01696de"
Jul 15 11:17:37.996001 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.995983    2566 scope.go:117] "RemoveContainer" containerID="4ddc087224c8873154073c3f9aa8922f475d2e77e4a55b5a4070df401bdd7f1d"
Jul 15 11:17:37.996270 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:17:37.996253    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ddc087224c8873154073c3f9aa8922f475d2e77e4a55b5a4070df401bdd7f1d\": container with ID starting with 4ddc087224c8873154073c3f9aa8922f475d2e77e4a55b5a4070df401bdd7f1d not found: ID does not exist" containerID="4ddc087224c8873154073c3f9aa8922f475d2e77e4a55b5a4070df401bdd7f1d"
Jul 15 11:17:37.996315 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.996281    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ddc087224c8873154073c3f9aa8922f475d2e77e4a55b5a4070df401bdd7f1d"} err="failed to get container status \"4ddc087224c8873154073c3f9aa8922f475d2e77e4a55b5a4070df401bdd7f1d\": rpc error: code = NotFound desc = could not find container \"4ddc087224c8873154073c3f9aa8922f475d2e77e4a55b5a4070df401bdd7f1d\": container with ID starting with 4ddc087224c8873154073c3f9aa8922f475d2e77e4a55b5a4070df401bdd7f1d not found: ID does not exist"
Jul 15 11:17:37.996315 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.996300    2566 scope.go:117] "RemoveContainer" containerID="69389544dfc0d7f10ce38cd72ae3432e1d611ef9bc9a92d6ee242951f01696de"
Jul 15 11:17:37.996929 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:17:37.996908    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69389544dfc0d7f10ce38cd72ae3432e1d611ef9bc9a92d6ee242951f01696de\": container with ID starting with 69389544dfc0d7f10ce38cd72ae3432e1d611ef9bc9a92d6ee242951f01696de not found: ID does not exist" containerID="69389544dfc0d7f10ce38cd72ae3432e1d611ef9bc9a92d6ee242951f01696de"
Jul 15 11:17:37.997015 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:37.996934    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69389544dfc0d7f10ce38cd72ae3432e1d611ef9bc9a92d6ee242951f01696de"} err="failed to get container status \"69389544dfc0d7f10ce38cd72ae3432e1d611ef9bc9a92d6ee242951f01696de\": rpc error: code = NotFound desc = could not find container \"69389544dfc0d7f10ce38cd72ae3432e1d611ef9bc9a92d6ee242951f01696de\": container with ID starting with 69389544dfc0d7f10ce38cd72ae3432e1d611ef9bc9a92d6ee242951f01696de not found: ID does not exist"
Jul 15 11:17:38.023667 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:38.023638    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"]
Jul 15 11:17:38.028629 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:38.028603    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-c9875db89-4wq4r"]
Jul 15 11:17:39.267189 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:39.267151    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" path="/var/lib/kubelet/pods/2225d2e8-6c95-4ee1-8f6c-ac61e41008ef/volumes"
Jul 15 11:17:54.404426 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.404391    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"]
Jul 15 11:17:54.405113 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.404754    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="main"
Jul 15 11:17:54.405113 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.404766    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="main"
Jul 15 11:17:54.405113 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.404775    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="storage-initializer"
Jul 15 11:17:54.405113 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.404780    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="storage-initializer"
Jul 15 11:17:54.405113 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.404843    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2225d2e8-6c95-4ee1-8f6c-ac61e41008ef" containerName="main"
Jul 15 11:17:54.407273 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.407252    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.410116 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.410095    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-9ttrh\""
Jul 15 11:17:54.410277 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.410172    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 15 11:17:54.418833 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.418810    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"]
Jul 15 11:17:54.420182 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.420157    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.420273 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.420191    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.420273 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.420207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.420273 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.420229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.420273 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.420245    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.420435 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.420287    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e189f69-4813-4d9e-a89f-b1411f26cf4c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.420435 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.420366    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5x7n\" (UniqueName: \"kubernetes.io/projected/3e189f69-4813-4d9e-a89f-b1411f26cf4c-kube-api-access-h5x7n\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.421469 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.421452    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.430123 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.430098    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"]
Jul 15 11:17:54.450232 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.450200    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"]
Jul 15 11:17:54.520840 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.520807    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.520840 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.520840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.521108 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.520861    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.521108 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.520881    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.521108 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.520905    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.521108 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.520970    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.521108 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.521010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.521108 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.521051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e189f69-4813-4d9e-a89f-b1411f26cf4c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.521398 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.521119    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.521398 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.521147    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq54g\" (UniqueName: \"kubernetes.io/projected/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-kube-api-access-zq54g\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.521398 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.521187    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5x7n\" (UniqueName: \"kubernetes.io/projected/3e189f69-4813-4d9e-a89f-b1411f26cf4c-kube-api-access-h5x7n\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.521398 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.521200    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.521398 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.521221    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.521398 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.521253    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.521398 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.521279    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.521398 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.521298    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.521398 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.521317    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.521718 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.521589    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.523465 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.523444    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.523679 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.523661    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e189f69-4813-4d9e-a89f-b1411f26cf4c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.530802 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.530781    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5x7n\" (UniqueName: \"kubernetes.io/projected/3e189f69-4813-4d9e-a89f-b1411f26cf4c-kube-api-access-h5x7n\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.621708 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.621678    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.621860 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.621712    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.621860 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.621742    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.621860 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.621811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.621860 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.621837    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq54g\" (UniqueName: \"kubernetes.io/projected/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-kube-api-access-zq54g\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.622036 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.621870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.622036 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.621907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.622144 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.622117    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.622239 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.622146    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.622309 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.622265    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.622379 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.622356    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.624181 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.624164    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.624334 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.624318    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.633928 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.633899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq54g\" (UniqueName: \"kubernetes.io/projected/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-kube-api-access-zq54g\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.719166 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.719063    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:54.732388 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.732147    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:17:54.882180 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.882139    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"]
Jul 15 11:17:54.891361 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.891341    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 11:17:54.913720 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:54.913698    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"]
Jul 15 11:17:54.916023 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:17:54.915999    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ecdfe01_3f62_49f7_a3fb_3914ba11df6c.slice/crio-12d939f702c47d528f5e7d03d78d328d423a43cc39ae3f896f4707d568e8f5a4 WatchSource:0}: Error finding container 12d939f702c47d528f5e7d03d78d328d423a43cc39ae3f896f4707d568e8f5a4: Status 404 returned error can't find the container with id 12d939f702c47d528f5e7d03d78d328d423a43cc39ae3f896f4707d568e8f5a4
Jul 15 11:17:55.078726 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:55.078688    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" event={"ID":"3e189f69-4813-4d9e-a89f-b1411f26cf4c","Type":"ContainerStarted","Data":"42b753d68888f3e0b22e4cb86b0b926e84ea7ba823bcfa148fb6fd5f2667acf1"}
Jul 15 11:17:55.080215 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:55.080189    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" event={"ID":"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c","Type":"ContainerStarted","Data":"2c3b1d3dc8ae64f977c5157b87a1c2e920acea1143a489ed74a53548a773e943"}
Jul 15 11:17:55.080382 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:55.080235    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" event={"ID":"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c","Type":"ContainerStarted","Data":"12d939f702c47d528f5e7d03d78d328d423a43cc39ae3f896f4707d568e8f5a4"}
Jul 15 11:17:57.901029 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:57.900990    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"]
Jul 15 11:17:58.097781 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:58.097748    2566 generic.go:356] "Generic (PLEG): container finished" podID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerID="2c3b1d3dc8ae64f977c5157b87a1c2e920acea1143a489ed74a53548a773e943" exitCode=0
Jul 15 11:17:58.097927 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:58.097823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" event={"ID":"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c","Type":"ContainerDied","Data":"2c3b1d3dc8ae64f977c5157b87a1c2e920acea1143a489ed74a53548a773e943"}
Jul 15 11:17:58.099446 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:58.099420    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" event={"ID":"3e189f69-4813-4d9e-a89f-b1411f26cf4c","Type":"ContainerStarted","Data":"dde3ce37e4201cd39e7385027677ed65ff42aec1882611adc60d8f625c693767"}
Jul 15 11:17:58.099591 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:58.099547    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:17:59.109608 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:59.109568    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" event={"ID":"3e189f69-4813-4d9e-a89f-b1411f26cf4c","Type":"ContainerStarted","Data":"4b9149b84d82e2d90498d1bdf073dcb12c5ffbdb75366e4136cfb98d4fbb854a"}
Jul 15 11:17:59.112483 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:59.112427    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" event={"ID":"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c","Type":"ContainerStarted","Data":"15dcee04ce2e67f75514f7397b1f8f88ae67dca9a03f1e9939c210af4a602075"}
Jul 15 11:17:59.159631 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:17:59.159575    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podStartSLOduration=2.116919729 podStartE2EDuration="5.159556444s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.042636715s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:17:59.159079499 +0000 UTC m=+1948.389047032" watchObservedRunningTime="2026-07-15 11:17:59.159556444 +0000 UTC m=+1948.389523976"
Jul 15 11:18:02.135102 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:02.135061    2566 generic.go:356] "Generic (PLEG): container finished" podID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerID="4b9149b84d82e2d90498d1bdf073dcb12c5ffbdb75366e4136cfb98d4fbb854a" exitCode=0
Jul 15 11:18:02.135579 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:02.135130    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" event={"ID":"3e189f69-4813-4d9e-a89f-b1411f26cf4c","Type":"ContainerDied","Data":"4b9149b84d82e2d90498d1bdf073dcb12c5ffbdb75366e4136cfb98d4fbb854a"}
Jul 15 11:18:03.143193 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:03.143157    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" event={"ID":"3e189f69-4813-4d9e-a89f-b1411f26cf4c","Type":"ContainerStarted","Data":"dc9654d199b6a69629c4e56d398dfb1bc83c671a0288f09c26d3b0a94ad332b4"}
Jul 15 11:18:03.169293 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:03.169232    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podStartSLOduration=3.724620115 podStartE2EDuration="9.169214561s" totalImagesPullingTime="2.510671787s" totalInitContainerRuntime="2.933922659s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:18:03.168366053 +0000 UTC m=+1952.398333583" watchObservedRunningTime="2026-07-15 11:18:03.169214561 +0000 UTC m=+1952.399182092"
Jul 15 11:18:04.719537 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:04.719475    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:18:04.719930 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:04.719558    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:18:04.721222 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:04.721189    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 15 11:18:04.733701 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:04.733678    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:18:04.733836 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:04.733741    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:18:04.735247 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:04.735219    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 15 11:18:12.930402 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:12.930359    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt" podUID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" containerName="main" containerID="cri-o://b4e7587019e92b2179330017bcd6e0ccdeb2d1e5e1d03a966121a69ac2b691e2" gracePeriod=45
Jul 15 11:18:12.981275 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:12.981229    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt" podUID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 15 11:18:13.586868 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.586845    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:18:13.699973 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.699938    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-dshm\") pod \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") "
Jul 15 11:18:13.700156 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.699989    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89e3c5b-09e9-43cd-a100-4960c89c1c84-tls-certs\") pod \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") "
Jul 15 11:18:13.700156 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.700030    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-model-cache\") pod \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") "
Jul 15 11:18:13.700156 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.700049    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-tmp-dir\") pod \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") "
Jul 15 11:18:13.700156 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.700065    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hkfq\" (UniqueName: \"kubernetes.io/projected/c89e3c5b-09e9-43cd-a100-4960c89c1c84-kube-api-access-9hkfq\") pod \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") "
Jul 15 11:18:13.700156 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.700096    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-kserve-provision-location\") pod \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") "
Jul 15 11:18:13.700156 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.700135    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c89e3c5b-09e9-43cd-a100-4960c89c1c84-cabundle-cert\") pod \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") "
Jul 15 11:18:13.700479 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.700164    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-home\") pod \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\" (UID: \"c89e3c5b-09e9-43cd-a100-4960c89c1c84\") "
Jul 15 11:18:13.700479 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.700388    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-model-cache" (OuterVolumeSpecName: "model-cache") pod "c89e3c5b-09e9-43cd-a100-4960c89c1c84" (UID: "c89e3c5b-09e9-43cd-a100-4960c89c1c84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:18:13.700636 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.700558    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c89e3c5b-09e9-43cd-a100-4960c89c1c84" (UID: "c89e3c5b-09e9-43cd-a100-4960c89c1c84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:18:13.700780 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.700735    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89e3c5b-09e9-43cd-a100-4960c89c1c84-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c89e3c5b-09e9-43cd-a100-4960c89c1c84" (UID: "c89e3c5b-09e9-43cd-a100-4960c89c1c84"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:18:13.700780 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.700750    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-home" (OuterVolumeSpecName: "home") pod "c89e3c5b-09e9-43cd-a100-4960c89c1c84" (UID: "c89e3c5b-09e9-43cd-a100-4960c89c1c84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:18:13.702922 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.702878    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-dshm" (OuterVolumeSpecName: "dshm") pod "c89e3c5b-09e9-43cd-a100-4960c89c1c84" (UID: "c89e3c5b-09e9-43cd-a100-4960c89c1c84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:18:13.702922 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.702899    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89e3c5b-09e9-43cd-a100-4960c89c1c84-kube-api-access-9hkfq" (OuterVolumeSpecName: "kube-api-access-9hkfq") pod "c89e3c5b-09e9-43cd-a100-4960c89c1c84" (UID: "c89e3c5b-09e9-43cd-a100-4960c89c1c84"). InnerVolumeSpecName "kube-api-access-9hkfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:18:13.703085 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.702999    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89e3c5b-09e9-43cd-a100-4960c89c1c84-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c89e3c5b-09e9-43cd-a100-4960c89c1c84" (UID: "c89e3c5b-09e9-43cd-a100-4960c89c1c84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:18:13.758560 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.758508    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c89e3c5b-09e9-43cd-a100-4960c89c1c84" (UID: "c89e3c5b-09e9-43cd-a100-4960c89c1c84"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:18:13.801154 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.801122    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c89e3c5b-09e9-43cd-a100-4960c89c1c84-cabundle-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:18:13.801154 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.801150    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:18:13.801154 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.801158    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:18:13.801376 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.801166    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89e3c5b-09e9-43cd-a100-4960c89c1c84-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:18:13.801376 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.801173    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:18:13.801376 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.801180    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:18:13.801376 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.801188    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hkfq\" (UniqueName: \"kubernetes.io/projected/c89e3c5b-09e9-43cd-a100-4960c89c1c84-kube-api-access-9hkfq\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:18:13.801376 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:13.801197    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c89e3c5b-09e9-43cd-a100-4960c89c1c84-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:18:14.206660 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:14.206624    2566 generic.go:356] "Generic (PLEG): container finished" podID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" containerID="b4e7587019e92b2179330017bcd6e0ccdeb2d1e5e1d03a966121a69ac2b691e2" exitCode=0
Jul 15 11:18:14.207071 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:14.206693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt" event={"ID":"c89e3c5b-09e9-43cd-a100-4960c89c1c84","Type":"ContainerDied","Data":"b4e7587019e92b2179330017bcd6e0ccdeb2d1e5e1d03a966121a69ac2b691e2"}
Jul 15 11:18:14.207071 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:14.206696    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"
Jul 15 11:18:14.207071 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:14.206783    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt" event={"ID":"c89e3c5b-09e9-43cd-a100-4960c89c1c84","Type":"ContainerDied","Data":"3fe1aa4d635e4f3baf174252ea8999ab59536c637ac297b684a677987dd184d2"}
Jul 15 11:18:14.207071 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:14.206815    2566 scope.go:117] "RemoveContainer" containerID="b4e7587019e92b2179330017bcd6e0ccdeb2d1e5e1d03a966121a69ac2b691e2"
Jul 15 11:18:14.217324 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:14.217308    2566 scope.go:117] "RemoveContainer" containerID="b2df56b6e13e9e9017bb42cd137f9ca9e55d346af3d1d890dcce358753451131"
Jul 15 11:18:14.225614 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:14.225591    2566 scope.go:117] "RemoveContainer" containerID="b4e7587019e92b2179330017bcd6e0ccdeb2d1e5e1d03a966121a69ac2b691e2"
Jul 15 11:18:14.225882 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:18:14.225862    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e7587019e92b2179330017bcd6e0ccdeb2d1e5e1d03a966121a69ac2b691e2\": container with ID starting with b4e7587019e92b2179330017bcd6e0ccdeb2d1e5e1d03a966121a69ac2b691e2 not found: ID does not exist" containerID="b4e7587019e92b2179330017bcd6e0ccdeb2d1e5e1d03a966121a69ac2b691e2"
Jul 15 11:18:14.225937 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:14.225890    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e7587019e92b2179330017bcd6e0ccdeb2d1e5e1d03a966121a69ac2b691e2"} err="failed to get container status \"b4e7587019e92b2179330017bcd6e0ccdeb2d1e5e1d03a966121a69ac2b691e2\": rpc error: code = NotFound desc = could not find container \"b4e7587019e92b2179330017bcd6e0ccdeb2d1e5e1d03a966121a69ac2b691e2\": container with ID starting with b4e7587019e92b2179330017bcd6e0ccdeb2d1e5e1d03a966121a69ac2b691e2 not found: ID does not exist"
Jul 15 11:18:14.225937 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:14.225907    2566 scope.go:117] "RemoveContainer" containerID="b2df56b6e13e9e9017bb42cd137f9ca9e55d346af3d1d890dcce358753451131"
Jul 15 11:18:14.226191 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:18:14.226173    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2df56b6e13e9e9017bb42cd137f9ca9e55d346af3d1d890dcce358753451131\": container with ID starting with b2df56b6e13e9e9017bb42cd137f9ca9e55d346af3d1d890dcce358753451131 not found: ID does not exist" containerID="b2df56b6e13e9e9017bb42cd137f9ca9e55d346af3d1d890dcce358753451131"
Jul 15 11:18:14.226239 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:14.226195    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2df56b6e13e9e9017bb42cd137f9ca9e55d346af3d1d890dcce358753451131"} err="failed to get container status \"b2df56b6e13e9e9017bb42cd137f9ca9e55d346af3d1d890dcce358753451131\": rpc error: code = NotFound desc = could not find container \"b2df56b6e13e9e9017bb42cd137f9ca9e55d346af3d1d890dcce358753451131\": container with ID starting with b2df56b6e13e9e9017bb42cd137f9ca9e55d346af3d1d890dcce358753451131 not found: ID does not exist"
Jul 15 11:18:14.230950 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:14.230928    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"]
Jul 15 11:18:14.237661 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:14.237636    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b654655c-4nnwt"]
Jul 15 11:18:14.720908 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:14.720762    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 15 11:18:14.732736 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:14.732714    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:18:14.733037 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:14.733001    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 15 11:18:15.268177 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:15.268140    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" path="/var/lib/kubelet/pods/c89e3c5b-09e9-43cd-a100-4960c89c1c84/volumes"
Jul 15 11:18:24.720003 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:24.719879    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 15 11:18:24.733797 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:24.733752    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 15 11:18:34.719803 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:34.719753    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 15 11:18:34.732996 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:34.732961    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 15 11:18:44.720253 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:44.720201    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 15 11:18:44.733840 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:44.733796    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 15 11:18:51.624777 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.624740    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"]
Jul 15 11:18:51.625168 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.625080    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" containerName="main"
Jul 15 11:18:51.625168 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.625091    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" containerName="main"
Jul 15 11:18:51.625168 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.625100    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" containerName="storage-initializer"
Jul 15 11:18:51.625168 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.625106    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" containerName="storage-initializer"
Jul 15 11:18:51.625303 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.625189    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c89e3c5b-09e9-43cd-a100-4960c89c1c84" containerName="main"
Jul 15 11:18:51.627744 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.627722    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.631766 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.631741    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 15 11:18:51.642271 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.642230    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"]
Jul 15 11:18:51.739161 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.739108    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.739161 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.739148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.739161 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.739171    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.739483 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.739274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.739483 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.739293    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.739483 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.739321    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f69l\" (UniqueName: \"kubernetes.io/projected/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-kube-api-access-9f69l\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.840022 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.839988    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.840022 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.840025    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.840299 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.840051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f69l\" (UniqueName: \"kubernetes.io/projected/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-kube-api-access-9f69l\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.840299 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.840218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.840299 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.840263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.840511 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.840305    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.840511 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.840453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.840644 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.840614    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.840701 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.840643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.842700 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.842673    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.842996 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.842975    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:51.849003 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:51.848976    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f69l\" (UniqueName: \"kubernetes.io/projected/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-kube-api-access-9f69l\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:18:54.719739 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:54.719689    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 15 11:18:54.733209 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:18:54.733166    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 15 11:19:01.831328 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:01.831278    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:19:01.842550 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:01.842520    2566 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:19:01.932881 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:01.932838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:19:01.933060 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:01.933031    2566 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:19:01.936374 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:01.936345    2566 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") DevicePath \"csi-28b39141b605ecccebeeed3c7d0a1bb248dc5f88d802eb0dabea42669c8e3558\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:19:02.038578 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:02.038548    2566 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4767870384ba5a33f8c589260e567a3f76dd02f928fb126f91966e97c3d5e812/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:19:02.055861 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:02.055826    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:19:02.140586 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:02.140481    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:19:02.299370 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:02.299335    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"]
Jul 15 11:19:02.303243 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:19:02.303211    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ba81ef_c196_4029_9f8e_c7651b2fd9e0.slice/crio-827fc98042d88cced511a09f7c4aa7f751b2e4d96aa1ffdec780db2da0f4ead9 WatchSource:0}: Error finding container 827fc98042d88cced511a09f7c4aa7f751b2e4d96aa1ffdec780db2da0f4ead9: Status 404 returned error can't find the container with id 827fc98042d88cced511a09f7c4aa7f751b2e4d96aa1ffdec780db2da0f4ead9
Jul 15 11:19:02.454844 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:02.454746    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" event={"ID":"56ba81ef-c196-4029-9f8e-c7651b2fd9e0","Type":"ContainerStarted","Data":"7a6055d3dc3c8348fdd8987a6381078d1faed3ce2b8294fbad7d80baadbf4067"}
Jul 15 11:19:02.454844 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:02.454818    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" event={"ID":"56ba81ef-c196-4029-9f8e-c7651b2fd9e0","Type":"ContainerStarted","Data":"827fc98042d88cced511a09f7c4aa7f751b2e4d96aa1ffdec780db2da0f4ead9"}
Jul 15 11:19:02.477414 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:02.477348    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" podStartSLOduration=11.477325957 podStartE2EDuration="11.477325957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:19:02.475592294 +0000 UTC m=+2011.705559825" watchObservedRunningTime="2026-07-15 11:19:02.477325957 +0000 UTC m=+2011.707293489"
Jul 15 11:19:03.141087 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:03.141043    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:19:04.720016 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:04.719965    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 15 11:19:04.733112 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:04.733062    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 15 11:19:12.141262 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:12.141217    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:19:12.143035 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:12.143001    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 15 11:19:14.719766 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:14.719717    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 15 11:19:14.733211 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:14.733169    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 15 11:19:22.141625 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:22.141570    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 15 11:19:24.719932 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:24.719872    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 15 11:19:24.733204 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:24.733160    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 15 11:19:32.141267 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:32.141212    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 15 11:19:34.720110 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:34.720061    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 15 11:19:34.733322 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:34.733277    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 15 11:19:42.141598 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:42.141520    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 15 11:19:44.720412 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:44.720363    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 15 11:19:44.733899 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:44.733859    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 15 11:19:52.141729 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:52.141635    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 15 11:19:54.720116 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:54.720068    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 15 11:19:54.733794 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:19:54.733752    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 15 11:20:02.141160 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:02.141104    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 15 11:20:04.720373 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:04.720317    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 15 11:20:04.733407 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:04.733360    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 15 11:20:12.141440 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:12.141380    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 15 11:20:14.720418 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:14.720366    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 15 11:20:14.733300 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:14.733267    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 15 11:20:22.141569 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:22.141518    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 15 11:20:24.720473 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:24.720426    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 15 11:20:24.733428 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:24.733395    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 15 11:20:32.141374 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:32.141325    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 15 11:20:34.720016 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:34.719954    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
Jul 15 11:20:34.733065 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:34.733021    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 15 11:20:38.310680 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:38.310648    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:20:38.315236 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:38.315208    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:20:42.141792 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:42.141755    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 15 11:20:44.735262 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:44.735229    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:20:44.743605 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:44.743580    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:20:44.751432 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:44.751402    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:20:44.752840 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:44.752815    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:20:52.141653 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:20:52.141603    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 15 11:21:00.893169 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:00.892183    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"]
Jul 15 11:21:00.896073 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:00.896043    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"]
Jul 15 11:21:02.141754 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:02.141705    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 15 11:21:12.141601 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:12.141549    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 15 11:21:15.945361 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:15.945292    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main" containerID="cri-o://15dcee04ce2e67f75514f7397b1f8f88ae67dca9a03f1e9939c210af4a602075" gracePeriod=45
Jul 15 11:21:15.947440 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:15.947406    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main" containerID="cri-o://dc9654d199b6a69629c4e56d398dfb1bc83c671a0288f09c26d3b0a94ad332b4" gracePeriod=45
Jul 15 11:21:17.061207 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.061165    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="llm-d-routing-sidecar" containerID="cri-o://dde3ce37e4201cd39e7385027677ed65ff42aec1882611adc60d8f625c693767" gracePeriod=44
Jul 15 11:21:17.071703 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:21:17.071672    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e189f69_4813_4d9e_a89f_b1411f26cf4c.slice/crio-dde3ce37e4201cd39e7385027677ed65ff42aec1882611adc60d8f625c693767.scope\": RecentStats: unable to find data in memory cache]"
Jul 15 11:21:17.147106 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.147075    2566 generic.go:356] "Generic (PLEG): container finished" podID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerID="15dcee04ce2e67f75514f7397b1f8f88ae67dca9a03f1e9939c210af4a602075" exitCode=0
Jul 15 11:21:17.147293 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.147128    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" event={"ID":"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c","Type":"ContainerDied","Data":"15dcee04ce2e67f75514f7397b1f8f88ae67dca9a03f1e9939c210af4a602075"}
Jul 15 11:21:17.149727 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.149686    2566 generic.go:356] "Generic (PLEG): container finished" podID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerID="dc9654d199b6a69629c4e56d398dfb1bc83c671a0288f09c26d3b0a94ad332b4" exitCode=0
Jul 15 11:21:17.149727 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.149716    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" event={"ID":"3e189f69-4813-4d9e-a89f-b1411f26cf4c","Type":"ContainerDied","Data":"dc9654d199b6a69629c4e56d398dfb1bc83c671a0288f09c26d3b0a94ad332b4"}
Jul 15 11:21:17.202233 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.202204    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:21:17.319441 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.319413    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:21:17.376127 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.376092    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-home\") pod \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") "
Jul 15 11:21:17.376127 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.376130    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq54g\" (UniqueName: \"kubernetes.io/projected/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-kube-api-access-zq54g\") pod \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") "
Jul 15 11:21:17.376377 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.376167    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-kserve-provision-location\") pod \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") "
Jul 15 11:21:17.376377 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.376262    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-model-cache\") pod \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") "
Jul 15 11:21:17.376377 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.376293    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-tls-certs\") pod \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") "
Jul 15 11:21:17.376377 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.376339    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-tmp-dir\") pod \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") "
Jul 15 11:21:17.376624 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.376385    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-dshm\") pod \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\" (UID: \"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c\") "
Jul 15 11:21:17.376624 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.376561    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" (UID: "0ecdfe01-3f62-49f7-a3fb-3914ba11df6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:21:17.376821 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.376804    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:21:17.379082 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.379049    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" (UID: "0ecdfe01-3f62-49f7-a3fb-3914ba11df6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:21:17.379208 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.379129    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-kube-api-access-zq54g" (OuterVolumeSpecName: "kube-api-access-zq54g") pod "0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" (UID: "0ecdfe01-3f62-49f7-a3fb-3914ba11df6c"). InnerVolumeSpecName "kube-api-access-zq54g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:21:17.379265 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.379210    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-home" (OuterVolumeSpecName: "home") pod "0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" (UID: "0ecdfe01-3f62-49f7-a3fb-3914ba11df6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:21:17.379265 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.379242    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-dshm" (OuterVolumeSpecName: "dshm") pod "0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" (UID: "0ecdfe01-3f62-49f7-a3fb-3914ba11df6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:21:17.389549 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.389514    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" (UID: "0ecdfe01-3f62-49f7-a3fb-3914ba11df6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:21:17.452056 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.452010    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" (UID: "0ecdfe01-3f62-49f7-a3fb-3914ba11df6c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:21:17.477590 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.477557    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e189f69-4813-4d9e-a89f-b1411f26cf4c-tls-certs\") pod \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") "
Jul 15 11:21:17.477803 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.477628    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-tmp-dir\") pod \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") "
Jul 15 11:21:17.477803 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.477653    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-home\") pod \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") "
Jul 15 11:21:17.477803 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.477678    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5x7n\" (UniqueName: \"kubernetes.io/projected/3e189f69-4813-4d9e-a89f-b1411f26cf4c-kube-api-access-h5x7n\") pod \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") "
Jul 15 11:21:17.477803 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.477708    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-dshm\") pod \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") "
Jul 15 11:21:17.477803 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.477751    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-model-cache\") pod \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") "
Jul 15 11:21:17.477803 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.477787    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-kserve-provision-location\") pod \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\" (UID: \"3e189f69-4813-4d9e-a89f-b1411f26cf4c\") "
Jul 15 11:21:17.478374 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.478327    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e189f69-4813-4d9e-a89f-b1411f26cf4c" (UID: "3e189f69-4813-4d9e-a89f-b1411f26cf4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:21:17.478782 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.478743    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-home" (OuterVolumeSpecName: "home") pod "3e189f69-4813-4d9e-a89f-b1411f26cf4c" (UID: "3e189f69-4813-4d9e-a89f-b1411f26cf4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:21:17.479076 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.478922    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:21:17.479076 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.478953    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:21:17.479076 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.478966    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:21:17.479076 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.478985    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:21:17.479076 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.478997    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:21:17.479076 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.479008    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:21:17.479386 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.479211    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq54g\" (UniqueName: \"kubernetes.io/projected/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-kube-api-access-zq54g\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:21:17.479386 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.479292    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:21:17.487459 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.481184    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e189f69-4813-4d9e-a89f-b1411f26cf4c-kube-api-access-h5x7n" (OuterVolumeSpecName: "kube-api-access-h5x7n") pod "3e189f69-4813-4d9e-a89f-b1411f26cf4c" (UID: "3e189f69-4813-4d9e-a89f-b1411f26cf4c"). InnerVolumeSpecName "kube-api-access-h5x7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:21:17.487459 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.481516    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e189f69-4813-4d9e-a89f-b1411f26cf4c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e189f69-4813-4d9e-a89f-b1411f26cf4c" (UID: "3e189f69-4813-4d9e-a89f-b1411f26cf4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:21:17.489176 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.489140    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-dshm" (OuterVolumeSpecName: "dshm") pod "3e189f69-4813-4d9e-a89f-b1411f26cf4c" (UID: "3e189f69-4813-4d9e-a89f-b1411f26cf4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:21:17.495739 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.495705    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e189f69-4813-4d9e-a89f-b1411f26cf4c" (UID: "3e189f69-4813-4d9e-a89f-b1411f26cf4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:21:17.557826 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.557767    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e189f69-4813-4d9e-a89f-b1411f26cf4c" (UID: "3e189f69-4813-4d9e-a89f-b1411f26cf4c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:21:17.579969 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.579937    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:21:17.579969 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.579965    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e189f69-4813-4d9e-a89f-b1411f26cf4c-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:21:17.579969 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.579977    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:21:17.580194 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.579986    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5x7n\" (UniqueName: \"kubernetes.io/projected/3e189f69-4813-4d9e-a89f-b1411f26cf4c-kube-api-access-h5x7n\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:21:17.580194 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:17.579994    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e189f69-4813-4d9e-a89f-b1411f26cf4c-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:21:18.156883 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.156851    2566 generic.go:356] "Generic (PLEG): container finished" podID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerID="dde3ce37e4201cd39e7385027677ed65ff42aec1882611adc60d8f625c693767" exitCode=0
Jul 15 11:21:18.157371 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.156920    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"
Jul 15 11:21:18.157371 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.156947    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" event={"ID":"3e189f69-4813-4d9e-a89f-b1411f26cf4c","Type":"ContainerDied","Data":"dde3ce37e4201cd39e7385027677ed65ff42aec1882611adc60d8f625c693767"}
Jul 15 11:21:18.157371 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.157058    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq" event={"ID":"3e189f69-4813-4d9e-a89f-b1411f26cf4c","Type":"ContainerDied","Data":"42b753d68888f3e0b22e4cb86b0b926e84ea7ba823bcfa148fb6fd5f2667acf1"}
Jul 15 11:21:18.157371 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.157103    2566 scope.go:117] "RemoveContainer" containerID="dc9654d199b6a69629c4e56d398dfb1bc83c671a0288f09c26d3b0a94ad332b4"
Jul 15 11:21:18.158848 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.158823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj" event={"ID":"0ecdfe01-3f62-49f7-a3fb-3914ba11df6c","Type":"ContainerDied","Data":"12d939f702c47d528f5e7d03d78d328d423a43cc39ae3f896f4707d568e8f5a4"}
Jul 15 11:21:18.158931 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.158892    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"
Jul 15 11:21:18.165753 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.165737    2566 scope.go:117] "RemoveContainer" containerID="4b9149b84d82e2d90498d1bdf073dcb12c5ffbdb75366e4136cfb98d4fbb854a"
Jul 15 11:21:18.177818 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.177793    2566 scope.go:117] "RemoveContainer" containerID="dde3ce37e4201cd39e7385027677ed65ff42aec1882611adc60d8f625c693767"
Jul 15 11:21:18.187834 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.187749    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"]
Jul 15 11:21:18.189270 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.189247    2566 scope.go:117] "RemoveContainer" containerID="dc9654d199b6a69629c4e56d398dfb1bc83c671a0288f09c26d3b0a94ad332b4"
Jul 15 11:21:18.189719 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:21:18.189689    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9654d199b6a69629c4e56d398dfb1bc83c671a0288f09c26d3b0a94ad332b4\": container with ID starting with dc9654d199b6a69629c4e56d398dfb1bc83c671a0288f09c26d3b0a94ad332b4 not found: ID does not exist" containerID="dc9654d199b6a69629c4e56d398dfb1bc83c671a0288f09c26d3b0a94ad332b4"
Jul 15 11:21:18.189833 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.189728    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9654d199b6a69629c4e56d398dfb1bc83c671a0288f09c26d3b0a94ad332b4"} err="failed to get container status \"dc9654d199b6a69629c4e56d398dfb1bc83c671a0288f09c26d3b0a94ad332b4\": rpc error: code = NotFound desc = could not find container \"dc9654d199b6a69629c4e56d398dfb1bc83c671a0288f09c26d3b0a94ad332b4\": container with ID starting with dc9654d199b6a69629c4e56d398dfb1bc83c671a0288f09c26d3b0a94ad332b4 not found: ID does not exist"
Jul 15 11:21:18.189833 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.189755    2566 scope.go:117] "RemoveContainer" containerID="4b9149b84d82e2d90498d1bdf073dcb12c5ffbdb75366e4136cfb98d4fbb854a"
Jul 15 11:21:18.190074 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:21:18.190053    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b9149b84d82e2d90498d1bdf073dcb12c5ffbdb75366e4136cfb98d4fbb854a\": container with ID starting with 4b9149b84d82e2d90498d1bdf073dcb12c5ffbdb75366e4136cfb98d4fbb854a not found: ID does not exist" containerID="4b9149b84d82e2d90498d1bdf073dcb12c5ffbdb75366e4136cfb98d4fbb854a"
Jul 15 11:21:18.190157 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.190080    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9149b84d82e2d90498d1bdf073dcb12c5ffbdb75366e4136cfb98d4fbb854a"} err="failed to get container status \"4b9149b84d82e2d90498d1bdf073dcb12c5ffbdb75366e4136cfb98d4fbb854a\": rpc error: code = NotFound desc = could not find container \"4b9149b84d82e2d90498d1bdf073dcb12c5ffbdb75366e4136cfb98d4fbb854a\": container with ID starting with 4b9149b84d82e2d90498d1bdf073dcb12c5ffbdb75366e4136cfb98d4fbb854a not found: ID does not exist"
Jul 15 11:21:18.190157 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.190100    2566 scope.go:117] "RemoveContainer" containerID="dde3ce37e4201cd39e7385027677ed65ff42aec1882611adc60d8f625c693767"
Jul 15 11:21:18.190557 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:21:18.190532    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde3ce37e4201cd39e7385027677ed65ff42aec1882611adc60d8f625c693767\": container with ID starting with dde3ce37e4201cd39e7385027677ed65ff42aec1882611adc60d8f625c693767 not found: ID does not exist" containerID="dde3ce37e4201cd39e7385027677ed65ff42aec1882611adc60d8f625c693767"
Jul 15 11:21:18.190669 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.190561    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde3ce37e4201cd39e7385027677ed65ff42aec1882611adc60d8f625c693767"} err="failed to get container status \"dde3ce37e4201cd39e7385027677ed65ff42aec1882611adc60d8f625c693767\": rpc error: code = NotFound desc = could not find container \"dde3ce37e4201cd39e7385027677ed65ff42aec1882611adc60d8f625c693767\": container with ID starting with dde3ce37e4201cd39e7385027677ed65ff42aec1882611adc60d8f625c693767 not found: ID does not exist"
Jul 15 11:21:18.190669 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.190583    2566 scope.go:117] "RemoveContainer" containerID="15dcee04ce2e67f75514f7397b1f8f88ae67dca9a03f1e9939c210af4a602075"
Jul 15 11:21:18.193266 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.193237    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-9c7bf686b-rbtgq"]
Jul 15 11:21:18.202611 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.202591    2566 scope.go:117] "RemoveContainer" containerID="2c3b1d3dc8ae64f977c5157b87a1c2e920acea1143a489ed74a53548a773e943"
Jul 15 11:21:18.203679 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.203653    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"]
Jul 15 11:21:18.208818 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:18.208795    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-7582nqj"]
Jul 15 11:21:19.267394 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:19.267355    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" path="/var/lib/kubelet/pods/0ecdfe01-3f62-49f7-a3fb-3914ba11df6c/volumes"
Jul 15 11:21:19.267906 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:19.267835    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" path="/var/lib/kubelet/pods/3e189f69-4813-4d9e-a89f-b1411f26cf4c/volumes"
Jul 15 11:21:22.151374 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:22.151295    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:21:22.159038 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:22.159014    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:21:29.974909 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.974876    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"]
Jul 15 11:21:29.975474 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.975197    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="llm-d-routing-sidecar"
Jul 15 11:21:29.975474 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.975207    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="llm-d-routing-sidecar"
Jul 15 11:21:29.975474 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.975227    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="storage-initializer"
Jul 15 11:21:29.975474 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.975232    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="storage-initializer"
Jul 15 11:21:29.975474 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.975239    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main"
Jul 15 11:21:29.975474 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.975245    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main"
Jul 15 11:21:29.975474 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.975251    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="storage-initializer"
Jul 15 11:21:29.975474 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.975258    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="storage-initializer"
Jul 15 11:21:29.975474 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.975266    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main"
Jul 15 11:21:29.975474 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.975271    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main"
Jul 15 11:21:29.975474 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.975318    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="main"
Jul 15 11:21:29.975474 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.975325    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ecdfe01-3f62-49f7-a3fb-3914ba11df6c" containerName="main"
Jul 15 11:21:29.975474 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.975331    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e189f69-4813-4d9e-a89f-b1411f26cf4c" containerName="llm-d-routing-sidecar"
Jul 15 11:21:29.980590 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.980557    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:29.982108 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.982085    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"]
Jul 15 11:21:29.983664 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.983637    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-9dtvw\""
Jul 15 11:21:29.983744 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.983689    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 11:21:29.983922 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.983909    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 15 11:21:29.985191 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.985173    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:29.985273 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.985239    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:29.985334 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.985294    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-dshm\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:29.985387 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.985353    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:29.985443 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.985386    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:29.985443 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.985424    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-model-cache\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:29.985592 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.985480    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-home\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:29.985592 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.985555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgshk\" (UniqueName: \"kubernetes.io/projected/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-kube-api-access-wgshk\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:29.993020 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.993000    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"]
Jul 15 11:21:29.999486 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:29.999452    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"]
Jul 15 11:21:30.086330 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:30.086330 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086331    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:30.086610 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086362    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq64k\" (UniqueName: \"kubernetes.io/projected/9e38acc6-5884-45a2-bb68-5b01b0c03279-kube-api-access-jq64k\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.086610 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086394    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-model-cache\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:30.086610 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086432    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e38acc6-5884-45a2-bb68-5b01b0c03279-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.086610 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086453    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-home\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:30.086610 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.086610 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086529    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgshk\" (UniqueName: \"kubernetes.io/projected/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-kube-api-access-wgshk\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:30.086610 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.086610 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:30.086610 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086594    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.086990 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086624    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.086990 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086654    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.086990 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-dshm\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:30.086990 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:30.086990 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-model-cache\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:30.086990 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086828    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-home\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:30.086990 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.086962    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:30.088839 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.088816    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-dshm\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:30.089056 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.089038    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:30.095563 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.095542    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgshk\" (UniqueName: \"kubernetes.io/projected/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-kube-api-access-wgshk\") pod \"custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:30.187403 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.187367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.187603 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.187419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.187603 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.187448    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.187603 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.187469    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.187603 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.187558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq64k\" (UniqueName: \"kubernetes.io/projected/9e38acc6-5884-45a2-bb68-5b01b0c03279-kube-api-access-jq64k\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.187831 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.187610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e38acc6-5884-45a2-bb68-5b01b0c03279-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.187831 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.187638    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.187831 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.187808    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.187988 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.187864    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.188049 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.187996    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.188049 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.188033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.190033 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.190012    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.190248 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.190231    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e38acc6-5884-45a2-bb68-5b01b0c03279-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.197046 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.197015    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq64k\" (UniqueName: \"kubernetes.io/projected/9e38acc6-5884-45a2-bb68-5b01b0c03279-kube-api-access-jq64k\") pod \"custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.291998 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.291966    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:30.300475 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.300450    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:30.460404 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.460342    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"]
Jul 15 11:21:30.464931 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:21:30.464904    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6da9a2_6ee3_4d8a_9d49_d7ae39701640.slice/crio-debd3e4a6108054fd78dfb2763c12388257c896330ead3fd24633f0e6825b35a WatchSource:0}: Error finding container debd3e4a6108054fd78dfb2763c12388257c896330ead3fd24633f0e6825b35a: Status 404 returned error can't find the container with id debd3e4a6108054fd78dfb2763c12388257c896330ead3fd24633f0e6825b35a
Jul 15 11:21:30.485759 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:30.485731    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"]
Jul 15 11:21:30.493673 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:21:30.493653    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e38acc6_5884_45a2_bb68_5b01b0c03279.slice/crio-197a8c1403bce07aa443e8329a0779f0acc0a16db4c35f92aef7be3c34e8596f WatchSource:0}: Error finding container 197a8c1403bce07aa443e8329a0779f0acc0a16db4c35f92aef7be3c34e8596f: Status 404 returned error can't find the container with id 197a8c1403bce07aa443e8329a0779f0acc0a16db4c35f92aef7be3c34e8596f
Jul 15 11:21:31.219905 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:31.219863    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" event={"ID":"9e38acc6-5884-45a2-bb68-5b01b0c03279","Type":"ContainerStarted","Data":"3d020bcd033a1473476266ec31b480cf1ef22acaa51026219d008a4a28dd0d5f"}
Jul 15 11:21:31.220393 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:31.219929    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" event={"ID":"9e38acc6-5884-45a2-bb68-5b01b0c03279","Type":"ContainerStarted","Data":"197a8c1403bce07aa443e8329a0779f0acc0a16db4c35f92aef7be3c34e8596f"}
Jul 15 11:21:31.221314 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:31.221293    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" event={"ID":"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640","Type":"ContainerStarted","Data":"dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020"}
Jul 15 11:21:31.221377 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:31.221333    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" event={"ID":"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640","Type":"ContainerStarted","Data":"debd3e4a6108054fd78dfb2763c12388257c896330ead3fd24633f0e6825b35a"}
Jul 15 11:21:31.221377 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:31.221356    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:32.242722 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:32.242675    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" event={"ID":"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640","Type":"ContainerStarted","Data":"6df5072bc6ce0a6360a3db46f4303b67c24749fe290185cf5224f5966b73a712"}
Jul 15 11:21:34.255251 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:34.255215    2566 generic.go:356] "Generic (PLEG): container finished" podID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerID="3d020bcd033a1473476266ec31b480cf1ef22acaa51026219d008a4a28dd0d5f" exitCode=0
Jul 15 11:21:34.255765 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:34.255297    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" event={"ID":"9e38acc6-5884-45a2-bb68-5b01b0c03279","Type":"ContainerDied","Data":"3d020bcd033a1473476266ec31b480cf1ef22acaa51026219d008a4a28dd0d5f"}
Jul 15 11:21:35.263372 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:35.263340    2566 generic.go:356] "Generic (PLEG): container finished" podID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerID="6df5072bc6ce0a6360a3db46f4303b67c24749fe290185cf5224f5966b73a712" exitCode=0
Jul 15 11:21:35.267967 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:35.267937    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" event={"ID":"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640","Type":"ContainerDied","Data":"6df5072bc6ce0a6360a3db46f4303b67c24749fe290185cf5224f5966b73a712"}
Jul 15 11:21:35.268113 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:35.267988    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" event={"ID":"9e38acc6-5884-45a2-bb68-5b01b0c03279","Type":"ContainerStarted","Data":"2abfe23934e8c7ee61c4fa52460f56fbe8b1d2b4ca60d4ad3b38fa46bdcac540"}
Jul 15 11:21:35.291965 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:35.291869    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podStartSLOduration=3.217711446 podStartE2EDuration="6.291847307s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.074135861s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:21:35.290132667 +0000 UTC m=+2164.520100197" watchObservedRunningTime="2026-07-15 11:21:35.291847307 +0000 UTC m=+2164.521814839"
Jul 15 11:21:36.276106 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:36.276069    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" event={"ID":"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640","Type":"ContainerStarted","Data":"f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c"}
Jul 15 11:21:36.315055 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:36.314990    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podStartSLOduration=3.873469148 podStartE2EDuration="7.314969775s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.441500627s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:21:36.309699084 +0000 UTC m=+2165.539666642" watchObservedRunningTime="2026-07-15 11:21:36.314969775 +0000 UTC m=+2165.544937307"
Jul 15 11:21:40.292328 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:40.292281    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:40.292857 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:40.292349    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:40.293989 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:40.293955    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:21:40.302288 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:40.301421    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:40.302288 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:40.301477    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:21:40.302704 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:40.302668    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:21:40.307953 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:40.307933    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:21:50.292881 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:50.292833    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:21:50.301782 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:50.301737    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:21:53.691931 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:21:53.691897    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"]
Jul 15 11:22:00.293790 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:00.293392    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:22:00.301813 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:00.301772    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:22:08.716506 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:08.716411    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main" containerID="cri-o://7a6055d3dc3c8348fdd8987a6381078d1faed3ce2b8294fbad7d80baadbf4067" gracePeriod=45
Jul 15 11:22:09.152850 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.152802    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 15 11:22:09.451451 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.451349    2566 generic.go:356] "Generic (PLEG): container finished" podID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerID="7a6055d3dc3c8348fdd8987a6381078d1faed3ce2b8294fbad7d80baadbf4067" exitCode=0
Jul 15 11:22:09.451451 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.451426    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" event={"ID":"56ba81ef-c196-4029-9f8e-c7651b2fd9e0","Type":"ContainerDied","Data":"7a6055d3dc3c8348fdd8987a6381078d1faed3ce2b8294fbad7d80baadbf4067"}
Jul 15 11:22:09.485009 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.484972    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:22:09.657529 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.657481    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-tls-certs\") pod \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") "
Jul 15 11:22:09.657529 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.657533    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-tmp-dir\") pod \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") "
Jul 15 11:22:09.657800 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.657558    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-home\") pod \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") "
Jul 15 11:22:09.657800 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.657600    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-model-cache\") pod \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") "
Jul 15 11:22:09.657800 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.657619    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-dshm\") pod \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") "
Jul 15 11:22:09.657800 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.657644    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f69l\" (UniqueName: \"kubernetes.io/projected/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-kube-api-access-9f69l\") pod \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") "
Jul 15 11:22:09.657800 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.657769    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\" (UID: \"56ba81ef-c196-4029-9f8e-c7651b2fd9e0\") "
Jul 15 11:22:09.658090 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.657852    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "56ba81ef-c196-4029-9f8e-c7651b2fd9e0" (UID: "56ba81ef-c196-4029-9f8e-c7651b2fd9e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:22:09.658090 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.658065    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:22:09.658334 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.658249    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-home" (OuterVolumeSpecName: "home") pod "56ba81ef-c196-4029-9f8e-c7651b2fd9e0" (UID: "56ba81ef-c196-4029-9f8e-c7651b2fd9e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:22:09.660481 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.660450    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-dshm" (OuterVolumeSpecName: "dshm") pod "56ba81ef-c196-4029-9f8e-c7651b2fd9e0" (UID: "56ba81ef-c196-4029-9f8e-c7651b2fd9e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:22:09.660890 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.660860    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56ba81ef-c196-4029-9f8e-c7651b2fd9e0" (UID: "56ba81ef-c196-4029-9f8e-c7651b2fd9e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:22:09.660997 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.660972    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-kube-api-access-9f69l" (OuterVolumeSpecName: "kube-api-access-9f69l") pod "56ba81ef-c196-4029-9f8e-c7651b2fd9e0" (UID: "56ba81ef-c196-4029-9f8e-c7651b2fd9e0"). InnerVolumeSpecName "kube-api-access-9f69l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:22:09.662893 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.662868    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4" (OuterVolumeSpecName: "kserve-pvc-source") pod "56ba81ef-c196-4029-9f8e-c7651b2fd9e0" (UID: "56ba81ef-c196-4029-9f8e-c7651b2fd9e0"). InnerVolumeSpecName "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 15 11:22:09.675567 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.675526    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56ba81ef-c196-4029-9f8e-c7651b2fd9e0" (UID: "56ba81ef-c196-4029-9f8e-c7651b2fd9e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:22:09.759517 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.759459    2566 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") on node \"ip-10-0-133-26.ec2.internal\" "
Jul 15 11:22:09.759994 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.759525    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:22:09.759994 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.759544    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:22:09.759994 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.759556    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:22:09.759994 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.759568    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:22:09.759994 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.759583    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f69l\" (UniqueName: \"kubernetes.io/projected/56ba81ef-c196-4029-9f8e-c7651b2fd9e0-kube-api-access-9f69l\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:22:09.792153 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.792132    2566 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4") on node "ip-10-0-133-26.ec2.internal"
Jul 15 11:22:09.860613 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:09.860576    2566 reconciler_common.go:299] "Volume detached for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"csi-28b39141b605ecccebeeed3c7d0a1bb248dc5f88d802eb0dabea42669c8e3558\""
Jul 15 11:22:10.292827 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:10.292759    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:22:10.300993 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:10.300955    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:22:10.458337 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:10.458302    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr" event={"ID":"56ba81ef-c196-4029-9f8e-c7651b2fd9e0","Type":"ContainerDied","Data":"827fc98042d88cced511a09f7c4aa7f751b2e4d96aa1ffdec780db2da0f4ead9"}
Jul 15 11:22:10.458577 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:10.458367    2566 scope.go:117] "RemoveContainer" containerID="7a6055d3dc3c8348fdd8987a6381078d1faed3ce2b8294fbad7d80baadbf4067"
Jul 15 11:22:10.458577 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:10.458316    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"
Jul 15 11:22:10.485233 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:10.485194    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"]
Jul 15 11:22:10.492970 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:10.492944    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dm7mhr"]
Jul 15 11:22:11.267454 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:11.267411    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" path="/var/lib/kubelet/pods/56ba81ef-c196-4029-9f8e-c7651b2fd9e0/volumes"
Jul 15 11:22:20.293217 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:20.293169    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:22:20.301408 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:20.301372    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:22:24.244755 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.244721    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"]
Jul 15 11:22:24.245329 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.245188    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main"
Jul 15 11:22:24.245329 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.245207    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main"
Jul 15 11:22:24.245461 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.245336    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="56ba81ef-c196-4029-9f8e-c7651b2fd9e0" containerName="main"
Jul 15 11:22:24.248558 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.248522    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.250851 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.250824    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"]
Jul 15 11:22:24.251380 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.251359    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 15 11:22:24.251513 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.251440    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-g2bzg\""
Jul 15 11:22:24.253186 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.253166    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.271945 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.271918    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"]
Jul 15 11:22:24.289218 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.289189    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.289358 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.289225    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.289358 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.289252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b465b4-239d-477d-a664-0b5af7e88404-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.289435 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.289385    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.289480 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.289439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.298666 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.298636    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"]
Jul 15 11:22:24.390156 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.390122    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h42gz\" (UniqueName: \"kubernetes.io/projected/0d7f3079-4c4f-4edb-af19-2364c351eb7c-kube-api-access-h42gz\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.390335 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.390175    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.390335 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.390230    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.390335 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.390259    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d7f3079-4c4f-4edb-af19-2364c351eb7c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.390335 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.390289    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.390335 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.390313    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b465b4-239d-477d-a664-0b5af7e88404-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.390669 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.390370    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.390669 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.390402    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2sc7\" (UniqueName: \"kubernetes.io/projected/64b465b4-239d-477d-a664-0b5af7e88404-kube-api-access-q2sc7\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.390669 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.390427    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.390669 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.390464    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.390669 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.390545    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.390669 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.390572    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.390669 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.390603    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.390985 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.390665    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.390985 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.390885    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.393007 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.392972    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.393220 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.393198    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b465b4-239d-477d-a664-0b5af7e88404-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.491720 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.491677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.491720 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.491722    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d7f3079-4c4f-4edb-af19-2364c351eb7c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.491975 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.491773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.491975 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.491801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2sc7\" (UniqueName: \"kubernetes.io/projected/64b465b4-239d-477d-a664-0b5af7e88404-kube-api-access-q2sc7\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.491975 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.491822    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.491975 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.491870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.491975 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.491942    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h42gz\" (UniqueName: \"kubernetes.io/projected/0d7f3079-4c4f-4edb-af19-2364c351eb7c-kube-api-access-h42gz\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.492242 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.492159    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.492434 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.492406    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.492605 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.492410    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.494332 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.494299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.495135 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.495085    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d7f3079-4c4f-4edb-af19-2364c351eb7c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:24.500936 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.500908    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2sc7\" (UniqueName: \"kubernetes.io/projected/64b465b4-239d-477d-a664-0b5af7e88404-kube-api-access-q2sc7\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:24.501079 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:24.500956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h42gz\" (UniqueName: \"kubernetes.io/projected/0d7f3079-4c4f-4edb-af19-2364c351eb7c-kube-api-access-h42gz\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:28.637611 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:28.637558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:28.643611 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:28.643585    2566 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:28.738839 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:28.738789    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:28.739034 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:28.738976    2566 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:28.741991 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:28.741959    2566 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") DevicePath \"csi-28b39141b605ecccebeeed3c7d0a1bb248dc5f88d802eb0dabea42669c8e3558\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:28.836097 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:28.836058    2566 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4767870384ba5a33f8c589260e567a3f76dd02f928fb126f91966e97c3d5e812/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:28.854532 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:28.854478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:28.941169 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:28.941057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:28.941357 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:28.941215    2566 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") DevicePath \"csi-28b39141b605ecccebeeed3c7d0a1bb248dc5f88d802eb0dabea42669c8e3558\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:28.944811 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:28.944781    2566 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") DevicePath \"csi-28b39141b605ecccebeeed3c7d0a1bb248dc5f88d802eb0dabea42669c8e3558\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:28.957111 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:28.957084    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:29.061851 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:29.061805    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:29.070347 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:29.070317    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:29.236277 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:29.235946    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"]
Jul 15 11:22:29.238796 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:22:29.238748    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7f3079_4c4f_4edb_af19_2364c351eb7c.slice/crio-380fc1fa5c0d10d23fd0993897a44686f25f4fd5f3b900ed175aa2d255ab557c WatchSource:0}: Error finding container 380fc1fa5c0d10d23fd0993897a44686f25f4fd5f3b900ed175aa2d255ab557c: Status 404 returned error can't find the container with id 380fc1fa5c0d10d23fd0993897a44686f25f4fd5f3b900ed175aa2d255ab557c
Jul 15 11:22:29.257462 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:29.257432    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"]
Jul 15 11:22:29.259773 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:22:29.259747    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b465b4_239d_477d_a664_0b5af7e88404.slice/crio-aa1b1af10147089f997e5243c3b4736a5a861b58b2ee06404df0030319dda08d WatchSource:0}: Error finding container aa1b1af10147089f997e5243c3b4736a5a861b58b2ee06404df0030319dda08d: Status 404 returned error can't find the container with id aa1b1af10147089f997e5243c3b4736a5a861b58b2ee06404df0030319dda08d
Jul 15 11:22:29.557232 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:29.557189    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" event={"ID":"64b465b4-239d-477d-a664-0b5af7e88404","Type":"ContainerStarted","Data":"7dfecd29ca7c77be0c9b810e24e214b1fb4a2639ef692ac7337c218776c1dd36"}
Jul 15 11:22:29.557461 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:29.557261    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" event={"ID":"64b465b4-239d-477d-a664-0b5af7e88404","Type":"ContainerStarted","Data":"aa1b1af10147089f997e5243c3b4736a5a861b58b2ee06404df0030319dda08d"}
Jul 15 11:22:29.559159 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:29.559122    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" event={"ID":"0d7f3079-4c4f-4edb-af19-2364c351eb7c","Type":"ContainerStarted","Data":"c4bc5d0ad24c0a50a2ebf9eb499f38324e920a5e4148e087993028d6f22bfe6d"}
Jul 15 11:22:29.559294 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:29.559183    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" event={"ID":"0d7f3079-4c4f-4edb-af19-2364c351eb7c","Type":"ContainerStarted","Data":"380fc1fa5c0d10d23fd0993897a44686f25f4fd5f3b900ed175aa2d255ab557c"}
Jul 15 11:22:29.559552 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:29.559527    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:29.577101 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:29.577034    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podStartSLOduration=5.577015557 podStartE2EDuration="5.577015557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:22:29.575987631 +0000 UTC m=+2218.805955161" watchObservedRunningTime="2026-07-15 11:22:29.577015557 +0000 UTC m=+2218.806983088"
Jul 15 11:22:30.292723 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:30.292676    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:22:30.301415 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:30.301367    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:22:30.569345 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:30.569243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" event={"ID":"0d7f3079-4c4f-4edb-af19-2364c351eb7c","Type":"ContainerStarted","Data":"5e3afcb619ed5b6588160af4351543affefeb5bd888ede4a19904de035867793"}
Jul 15 11:22:30.594699 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:30.594636    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podStartSLOduration=6.594615785 podStartE2EDuration="6.594615785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:22:30.591628244 +0000 UTC m=+2219.821595816" watchObservedRunningTime="2026-07-15 11:22:30.594615785 +0000 UTC m=+2219.824583318"
Jul 15 11:22:39.062277 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:39.062241    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:39.064338 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:39.064269    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:39.064595 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:39.064566    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:22:39.071218 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:39.071192    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:39.071918 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:39.071896    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:22:39.072333 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:39.072304    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:22:39.084357 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:39.084333    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:22:40.293949 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:40.293509    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:22:40.301659 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:40.301619    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:22:49.063319 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:49.063267    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:22:49.071416 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:49.071373    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:22:50.293403 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:50.293321    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:22:50.301790 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:50.301755    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:22:59.063441 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:59.063319    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:22:59.071433 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:22:59.071395    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:23:00.293058 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:00.292594    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:23:00.301717 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:00.301672    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:23:09.063278 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:09.063232    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:23:09.071444 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:09.071408    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:23:10.293416 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:10.293367    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:23:10.300932 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:10.300890    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:23:19.063116 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:19.063070    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:23:19.071452 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:19.071414    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:23:20.292714 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:20.292625    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:23:20.301350 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:20.301307    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:23:29.063519 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:29.063348    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:23:29.071840 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:29.071742    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:23:30.293206 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:30.293164    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:23:30.301077 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:30.301033    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:23:39.063366 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:39.063310    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:23:39.071453 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:39.071408    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:23:40.292441 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:40.292392    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:23:40.301294 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:40.301252    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:23:49.062534 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:49.062470    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:23:49.070801 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:49.070760    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:23:50.292657 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:50.292600    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:23:50.301150 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:50.301118    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:23:59.062617 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:59.062574    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:23:59.070901 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:23:59.070867    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:24:00.293077 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:00.293018    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:24:00.301771 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:00.301737    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:24:09.063184 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:09.063116    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:24:09.070824 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:09.070787    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:24:10.293230 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:10.293182    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:24:10.301761 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:10.301726    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:24:19.063197 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:19.063148    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:24:19.071657 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:19.071625    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:24:20.293284 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:20.293232    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:24:20.301447 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:20.301413    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:24:29.062821 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:29.062776    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:24:29.071589 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:29.071553    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:24:30.293361 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:30.293310    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:24:30.301589 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:30.301555    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:24:39.062925 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:39.062876    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:24:39.071655 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:39.071619    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:24:40.293237 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:40.292887    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8001/health\": dial tcp 10.133.0.47:8001: connect: connection refused"
Jul 15 11:24:40.301438 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:40.301392    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 15 11:24:49.063022 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:49.062978    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:24:49.070772 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:49.070730    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:24:50.309095 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:50.309057    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:24:50.312340 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:50.312310    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:24:50.321082 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:50.321040    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:24:50.321339 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:50.321313    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:24:59.062386 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:59.062340    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:24:59.071605 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:24:59.071575    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:25:09.062817 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:09.062763    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:25:09.070664 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:09.070632    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:25:19.062437 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:19.062396    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:25:19.071208 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:19.071175    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:25:29.062806 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:29.062758    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Jul 15 11:25:29.071572 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:29.071539    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 15 11:25:38.345329 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:38.345301    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:25:38.350904 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:38.350882    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:25:39.077225 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:39.077194    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:25:39.080784 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:39.080759    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:25:39.090197 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:39.090170    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:25:39.090346 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:39.090244    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:25:44.795104 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:44.795066    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"]
Jul 15 11:25:44.797027 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:44.797002    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"]
Jul 15 11:25:59.197719 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:59.197638    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"]
Jul 15 11:25:59.205534 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:59.205487    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"]
Jul 15 11:25:59.820300 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:59.820263    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main" containerID="cri-o://5e3afcb619ed5b6588160af4351543affefeb5bd888ede4a19904de035867793" gracePeriod=45
Jul 15 11:25:59.820949 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:25:59.820771    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main" containerID="cri-o://7dfecd29ca7c77be0c9b810e24e214b1fb4a2639ef692ac7337c218776c1dd36" gracePeriod=45
Jul 15 11:26:00.631300 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.631238    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="llm-d-routing-sidecar" containerID="cri-o://c4bc5d0ad24c0a50a2ebf9eb499f38324e920a5e4148e087993028d6f22bfe6d" gracePeriod=45
Jul 15 11:26:00.674059 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.674027    2566 generic.go:356] "Generic (PLEG): container finished" podID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerID="5e3afcb619ed5b6588160af4351543affefeb5bd888ede4a19904de035867793" exitCode=0
Jul 15 11:26:00.674251 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.674099    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" event={"ID":"0d7f3079-4c4f-4edb-af19-2364c351eb7c","Type":"ContainerDied","Data":"5e3afcb619ed5b6588160af4351543affefeb5bd888ede4a19904de035867793"}
Jul 15 11:26:00.675937 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.675913    2566 generic.go:356] "Generic (PLEG): container finished" podID="64b465b4-239d-477d-a664-0b5af7e88404" containerID="7dfecd29ca7c77be0c9b810e24e214b1fb4a2639ef692ac7337c218776c1dd36" exitCode=0
Jul 15 11:26:00.676057 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.675959    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" event={"ID":"64b465b4-239d-477d-a664-0b5af7e88404","Type":"ContainerDied","Data":"7dfecd29ca7c77be0c9b810e24e214b1fb4a2639ef692ac7337c218776c1dd36"}
Jul 15 11:26:00.776560 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.776534    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:26:00.819738 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.819712    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2sc7\" (UniqueName: \"kubernetes.io/projected/64b465b4-239d-477d-a664-0b5af7e88404-kube-api-access-q2sc7\") pod \"64b465b4-239d-477d-a664-0b5af7e88404\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") "
Jul 15 11:26:00.819895 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.819817    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"64b465b4-239d-477d-a664-0b5af7e88404\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") "
Jul 15 11:26:00.819895 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.819845    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-tmp-dir\") pod \"64b465b4-239d-477d-a664-0b5af7e88404\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") "
Jul 15 11:26:00.819895 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.819867    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-home\") pod \"64b465b4-239d-477d-a664-0b5af7e88404\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") "
Jul 15 11:26:00.820064 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.819934    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b465b4-239d-477d-a664-0b5af7e88404-tls-certs\") pod \"64b465b4-239d-477d-a664-0b5af7e88404\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") "
Jul 15 11:26:00.820064 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.819984    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-model-cache\") pod \"64b465b4-239d-477d-a664-0b5af7e88404\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") "
Jul 15 11:26:00.820064 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.820007    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-dshm\") pod \"64b465b4-239d-477d-a664-0b5af7e88404\" (UID: \"64b465b4-239d-477d-a664-0b5af7e88404\") "
Jul 15 11:26:00.820559 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.820532    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-model-cache" (OuterVolumeSpecName: "model-cache") pod "64b465b4-239d-477d-a664-0b5af7e88404" (UID: "64b465b4-239d-477d-a664-0b5af7e88404"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:00.822250 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.821350    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-home" (OuterVolumeSpecName: "home") pod "64b465b4-239d-477d-a664-0b5af7e88404" (UID: "64b465b4-239d-477d-a664-0b5af7e88404"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:00.822795 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.822771    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-dshm" (OuterVolumeSpecName: "dshm") pod "64b465b4-239d-477d-a664-0b5af7e88404" (UID: "64b465b4-239d-477d-a664-0b5af7e88404"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:00.822905 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.822797    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b465b4-239d-477d-a664-0b5af7e88404-kube-api-access-q2sc7" (OuterVolumeSpecName: "kube-api-access-q2sc7") pod "64b465b4-239d-477d-a664-0b5af7e88404" (UID: "64b465b4-239d-477d-a664-0b5af7e88404"). InnerVolumeSpecName "kube-api-access-q2sc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:26:00.822905 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.822883    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64b465b4-239d-477d-a664-0b5af7e88404-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64b465b4-239d-477d-a664-0b5af7e88404" (UID: "64b465b4-239d-477d-a664-0b5af7e88404"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:26:00.823200 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.823181    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4" (OuterVolumeSpecName: "kserve-pvc-source") pod "64b465b4-239d-477d-a664-0b5af7e88404" (UID: "64b465b4-239d-477d-a664-0b5af7e88404"). InnerVolumeSpecName "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 15 11:26:00.834674 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.834653    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64b465b4-239d-477d-a664-0b5af7e88404" (UID: "64b465b4-239d-477d-a664-0b5af7e88404"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:00.883125 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.883070    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:26:00.920612 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.920587    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-tmp-dir\") pod \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") "
Jul 15 11:26:00.920738 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.920632    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h42gz\" (UniqueName: \"kubernetes.io/projected/0d7f3079-4c4f-4edb-af19-2364c351eb7c-kube-api-access-h42gz\") pod \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") "
Jul 15 11:26:00.920738 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.920648    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-model-cache\") pod \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") "
Jul 15 11:26:00.920738 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.920679    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-dshm\") pod \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") "
Jul 15 11:26:00.920922 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.920790    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") "
Jul 15 11:26:00.920922 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.920823    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-home\") pod \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") "
Jul 15 11:26:00.920922 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.920864    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d7f3079-4c4f-4edb-af19-2364c351eb7c-tls-certs\") pod \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\" (UID: \"0d7f3079-4c4f-4edb-af19-2364c351eb7c\") "
Jul 15 11:26:00.921084 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.920993    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d7f3079-4c4f-4edb-af19-2364c351eb7c" (UID: "0d7f3079-4c4f-4edb-af19-2364c351eb7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:00.921195 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.921177    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:00.921280 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.921201    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:00.921280 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.921217    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:00.921280 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.921232    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2sc7\" (UniqueName: \"kubernetes.io/projected/64b465b4-239d-477d-a664-0b5af7e88404-kube-api-access-q2sc7\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:00.921280 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.921247    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:00.921280 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.921259    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64b465b4-239d-477d-a664-0b5af7e88404-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:00.921280 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.921269    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b465b4-239d-477d-a664-0b5af7e88404-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:00.921806 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.921778    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-home" (OuterVolumeSpecName: "home") pod "0d7f3079-4c4f-4edb-af19-2364c351eb7c" (UID: "0d7f3079-4c4f-4edb-af19-2364c351eb7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:00.923162 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.923142    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-dshm" (OuterVolumeSpecName: "dshm") pod "0d7f3079-4c4f-4edb-af19-2364c351eb7c" (UID: "0d7f3079-4c4f-4edb-af19-2364c351eb7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:00.923874 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.923849    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7f3079-4c4f-4edb-af19-2364c351eb7c-kube-api-access-h42gz" (OuterVolumeSpecName: "kube-api-access-h42gz") pod "0d7f3079-4c4f-4edb-af19-2364c351eb7c" (UID: "0d7f3079-4c4f-4edb-af19-2364c351eb7c"). InnerVolumeSpecName "kube-api-access-h42gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:26:00.923977 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.923872    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7f3079-4c4f-4edb-af19-2364c351eb7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d7f3079-4c4f-4edb-af19-2364c351eb7c" (UID: "0d7f3079-4c4f-4edb-af19-2364c351eb7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:26:00.924282 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.924255    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4" (OuterVolumeSpecName: "kserve-pvc-source") pod "0d7f3079-4c4f-4edb-af19-2364c351eb7c" (UID: "0d7f3079-4c4f-4edb-af19-2364c351eb7c"). InnerVolumeSpecName "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 15 11:26:00.934784 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:00.934762    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d7f3079-4c4f-4edb-af19-2364c351eb7c" (UID: "0d7f3079-4c4f-4edb-af19-2364c351eb7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:01.022210 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.022175    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:01.022210 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.022205    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h42gz\" (UniqueName: \"kubernetes.io/projected/0d7f3079-4c4f-4edb-af19-2364c351eb7c-kube-api-access-h42gz\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:01.022210 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.022216    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:01.022445 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.022246    2566 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") on node \"ip-10-0-133-26.ec2.internal\" "
Jul 15 11:26:01.022445 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.022255    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d7f3079-4c4f-4edb-af19-2364c351eb7c-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:01.022445 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.022264    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d7f3079-4c4f-4edb-af19-2364c351eb7c-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:01.049724 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.049706    2566 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4") on node "ip-10-0-133-26.ec2.internal"
Jul 15 11:26:01.123450 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.123410    2566 reconciler_common.go:299] "Volume detached for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"csi-28b39141b605ecccebeeed3c7d0a1bb248dc5f88d802eb0dabea42669c8e3558\""
Jul 15 11:26:01.681615 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.681581    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"
Jul 15 11:26:01.682079 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.681579    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn" event={"ID":"64b465b4-239d-477d-a664-0b5af7e88404","Type":"ContainerDied","Data":"aa1b1af10147089f997e5243c3b4736a5a861b58b2ee06404df0030319dda08d"}
Jul 15 11:26:01.682079 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.681720    2566 scope.go:117] "RemoveContainer" containerID="7dfecd29ca7c77be0c9b810e24e214b1fb4a2639ef692ac7337c218776c1dd36"
Jul 15 11:26:01.683254 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.683234    2566 generic.go:356] "Generic (PLEG): container finished" podID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerID="c4bc5d0ad24c0a50a2ebf9eb499f38324e920a5e4148e087993028d6f22bfe6d" exitCode=0
Jul 15 11:26:01.683356 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.683287    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"
Jul 15 11:26:01.683356 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.683294    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" event={"ID":"0d7f3079-4c4f-4edb-af19-2364c351eb7c","Type":"ContainerDied","Data":"c4bc5d0ad24c0a50a2ebf9eb499f38324e920a5e4148e087993028d6f22bfe6d"}
Jul 15 11:26:01.683356 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.683327    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l" event={"ID":"0d7f3079-4c4f-4edb-af19-2364c351eb7c","Type":"ContainerDied","Data":"380fc1fa5c0d10d23fd0993897a44686f25f4fd5f3b900ed175aa2d255ab557c"}
Jul 15 11:26:01.690666 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.690641    2566 scope.go:117] "RemoveContainer" containerID="5e3afcb619ed5b6588160af4351543affefeb5bd888ede4a19904de035867793"
Jul 15 11:26:01.698648 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.698621    2566 scope.go:117] "RemoveContainer" containerID="c4bc5d0ad24c0a50a2ebf9eb499f38324e920a5e4148e087993028d6f22bfe6d"
Jul 15 11:26:01.706117 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.706095    2566 scope.go:117] "RemoveContainer" containerID="5e3afcb619ed5b6588160af4351543affefeb5bd888ede4a19904de035867793"
Jul 15 11:26:01.706379 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:26:01.706360    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e3afcb619ed5b6588160af4351543affefeb5bd888ede4a19904de035867793\": container with ID starting with 5e3afcb619ed5b6588160af4351543affefeb5bd888ede4a19904de035867793 not found: ID does not exist" containerID="5e3afcb619ed5b6588160af4351543affefeb5bd888ede4a19904de035867793"
Jul 15 11:26:01.706447 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.706386    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3afcb619ed5b6588160af4351543affefeb5bd888ede4a19904de035867793"} err="failed to get container status \"5e3afcb619ed5b6588160af4351543affefeb5bd888ede4a19904de035867793\": rpc error: code = NotFound desc = could not find container \"5e3afcb619ed5b6588160af4351543affefeb5bd888ede4a19904de035867793\": container with ID starting with 5e3afcb619ed5b6588160af4351543affefeb5bd888ede4a19904de035867793 not found: ID does not exist"
Jul 15 11:26:01.706447 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.706404    2566 scope.go:117] "RemoveContainer" containerID="c4bc5d0ad24c0a50a2ebf9eb499f38324e920a5e4148e087993028d6f22bfe6d"
Jul 15 11:26:01.706661 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:26:01.706636    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4bc5d0ad24c0a50a2ebf9eb499f38324e920a5e4148e087993028d6f22bfe6d\": container with ID starting with c4bc5d0ad24c0a50a2ebf9eb499f38324e920a5e4148e087993028d6f22bfe6d not found: ID does not exist" containerID="c4bc5d0ad24c0a50a2ebf9eb499f38324e920a5e4148e087993028d6f22bfe6d"
Jul 15 11:26:01.706782 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.706668    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4bc5d0ad24c0a50a2ebf9eb499f38324e920a5e4148e087993028d6f22bfe6d"} err="failed to get container status \"c4bc5d0ad24c0a50a2ebf9eb499f38324e920a5e4148e087993028d6f22bfe6d\": rpc error: code = NotFound desc = could not find container \"c4bc5d0ad24c0a50a2ebf9eb499f38324e920a5e4148e087993028d6f22bfe6d\": container with ID starting with c4bc5d0ad24c0a50a2ebf9eb499f38324e920a5e4148e087993028d6f22bfe6d not found: ID does not exist"
Jul 15 11:26:01.709438 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.709415    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"]
Jul 15 11:26:01.716395 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.716374    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-6vz9l"]
Jul 15 11:26:01.730698 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.730670    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"]
Jul 15 11:26:01.735468 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:01.735439    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7949vpn"]
Jul 15 11:26:03.267982 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:03.267951    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" path="/var/lib/kubelet/pods/0d7f3079-4c4f-4edb-af19-2364c351eb7c/volumes"
Jul 15 11:26:03.268393 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:03.268379    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64b465b4-239d-477d-a664-0b5af7e88404" path="/var/lib/kubelet/pods/64b465b4-239d-477d-a664-0b5af7e88404/volumes"
Jul 15 11:26:14.223411 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:14.223330    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main" containerID="cri-o://2abfe23934e8c7ee61c4fa52460f56fbe8b1d2b4ca60d4ad3b38fa46bdcac540" gracePeriod=45
Jul 15 11:26:14.229398 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:14.229360    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main" containerID="cri-o://f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c" gracePeriod=45
Jul 15 11:26:14.960782 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:14.960728    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="llm-d-routing-sidecar" containerID="cri-o://dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020" gracePeriod=45
Jul 15 11:26:15.225967 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.225945    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:26:15.228810 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.228794    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:26:15.335298 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.335265    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-kserve-provision-location\") pod \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") "
Jul 15 11:26:15.335469 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.335307    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-home\") pod \"9e38acc6-5884-45a2-bb68-5b01b0c03279\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") "
Jul 15 11:26:15.335469 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.335325    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-tls-certs\") pod \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") "
Jul 15 11:26:15.335469 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.335343    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-home\") pod \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") "
Jul 15 11:26:15.335469 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.335366    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jq64k\" (UniqueName: \"kubernetes.io/projected/9e38acc6-5884-45a2-bb68-5b01b0c03279-kube-api-access-jq64k\") pod \"9e38acc6-5884-45a2-bb68-5b01b0c03279\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") "
Jul 15 11:26:15.335469 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.335384    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e38acc6-5884-45a2-bb68-5b01b0c03279-tls-certs\") pod \"9e38acc6-5884-45a2-bb68-5b01b0c03279\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") "
Jul 15 11:26:15.335469 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.335445    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-model-cache\") pod \"9e38acc6-5884-45a2-bb68-5b01b0c03279\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") "
Jul 15 11:26:15.335836 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.335484    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-kserve-provision-location\") pod \"9e38acc6-5884-45a2-bb68-5b01b0c03279\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") "
Jul 15 11:26:15.335836 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.335611    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-dshm\") pod \"9e38acc6-5884-45a2-bb68-5b01b0c03279\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") "
Jul 15 11:26:15.335836 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.335640    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgshk\" (UniqueName: \"kubernetes.io/projected/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-kube-api-access-wgshk\") pod \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") "
Jul 15 11:26:15.335836 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.335709    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-model-cache\") pod \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") "
Jul 15 11:26:15.335836 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.335731    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-tmp-dir\") pod \"9e38acc6-5884-45a2-bb68-5b01b0c03279\" (UID: \"9e38acc6-5884-45a2-bb68-5b01b0c03279\") "
Jul 15 11:26:15.335836 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.335772    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-tmp-dir\") pod \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") "
Jul 15 11:26:15.335836 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.335795    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-dshm\") pod \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\" (UID: \"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640\") "
Jul 15 11:26:15.336179 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.335982    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e38acc6-5884-45a2-bb68-5b01b0c03279" (UID: "9e38acc6-5884-45a2-bb68-5b01b0c03279"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:15.336179 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.336068    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-home" (OuterVolumeSpecName: "home") pod "9e38acc6-5884-45a2-bb68-5b01b0c03279" (UID: "9e38acc6-5884-45a2-bb68-5b01b0c03279"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:15.336179 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.336086    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-home" (OuterVolumeSpecName: "home") pod "7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" (UID: "7c6da9a2-6ee3-4d8a-9d49-d7ae39701640"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:15.336335 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.336129    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:15.336335 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.336235    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" (UID: "7c6da9a2-6ee3-4d8a-9d49-d7ae39701640"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:15.338274 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.338228    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-dshm" (OuterVolumeSpecName: "dshm") pod "7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" (UID: "7c6da9a2-6ee3-4d8a-9d49-d7ae39701640"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:15.338746 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.338699    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" (UID: "7c6da9a2-6ee3-4d8a-9d49-d7ae39701640"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:26:15.339790 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.339747    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e38acc6-5884-45a2-bb68-5b01b0c03279-kube-api-access-jq64k" (OuterVolumeSpecName: "kube-api-access-jq64k") pod "9e38acc6-5884-45a2-bb68-5b01b0c03279" (UID: "9e38acc6-5884-45a2-bb68-5b01b0c03279"). InnerVolumeSpecName "kube-api-access-jq64k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:26:15.340962 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.340906    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e38acc6-5884-45a2-bb68-5b01b0c03279-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e38acc6-5884-45a2-bb68-5b01b0c03279" (UID: "9e38acc6-5884-45a2-bb68-5b01b0c03279"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:26:15.340962 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.340912    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-dshm" (OuterVolumeSpecName: "dshm") pod "9e38acc6-5884-45a2-bb68-5b01b0c03279" (UID: "9e38acc6-5884-45a2-bb68-5b01b0c03279"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:15.340962 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.340915    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-kube-api-access-wgshk" (OuterVolumeSpecName: "kube-api-access-wgshk") pod "7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" (UID: "7c6da9a2-6ee3-4d8a-9d49-d7ae39701640"). InnerVolumeSpecName "kube-api-access-wgshk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:26:15.356580 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.356554    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" (UID: "7c6da9a2-6ee3-4d8a-9d49-d7ae39701640"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:15.356679 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.356603    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e38acc6-5884-45a2-bb68-5b01b0c03279" (UID: "9e38acc6-5884-45a2-bb68-5b01b0c03279"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:15.384185 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.384154    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9e38acc6-5884-45a2-bb68-5b01b0c03279" (UID: "9e38acc6-5884-45a2-bb68-5b01b0c03279"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:15.407237 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.407208    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" (UID: "7c6da9a2-6ee3-4d8a-9d49-d7ae39701640"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:26:15.436881 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.436859    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:15.436881 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.436881    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:15.436983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.436892    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq64k\" (UniqueName: \"kubernetes.io/projected/9e38acc6-5884-45a2-bb68-5b01b0c03279-kube-api-access-jq64k\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:15.436983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.436902    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e38acc6-5884-45a2-bb68-5b01b0c03279-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:15.436983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.436912    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:15.436983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.436919    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:15.436983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.436928    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgshk\" (UniqueName: \"kubernetes.io/projected/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-kube-api-access-wgshk\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:15.436983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.436935    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:15.436983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.436943    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:15.436983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.436950    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:15.436983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.436957    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:15.436983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.436964    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:15.436983 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.436972    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e38acc6-5884-45a2-bb68-5b01b0c03279-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:26:15.749147 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.749116    2566 generic.go:356] "Generic (PLEG): container finished" podID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerID="2abfe23934e8c7ee61c4fa52460f56fbe8b1d2b4ca60d4ad3b38fa46bdcac540" exitCode=0
Jul 15 11:26:15.749329 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.749169    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"
Jul 15 11:26:15.749329 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.749203    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" event={"ID":"9e38acc6-5884-45a2-bb68-5b01b0c03279","Type":"ContainerDied","Data":"2abfe23934e8c7ee61c4fa52460f56fbe8b1d2b4ca60d4ad3b38fa46bdcac540"}
Jul 15 11:26:15.749329 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.749262    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk" event={"ID":"9e38acc6-5884-45a2-bb68-5b01b0c03279","Type":"ContainerDied","Data":"197a8c1403bce07aa443e8329a0779f0acc0a16db4c35f92aef7be3c34e8596f"}
Jul 15 11:26:15.749329 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.749294    2566 scope.go:117] "RemoveContainer" containerID="2abfe23934e8c7ee61c4fa52460f56fbe8b1d2b4ca60d4ad3b38fa46bdcac540"
Jul 15 11:26:15.751360 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.751329    2566 generic.go:356] "Generic (PLEG): container finished" podID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerID="f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c" exitCode=0
Jul 15 11:26:15.751360 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.751360    2566 generic.go:356] "Generic (PLEG): container finished" podID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerID="dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020" exitCode=0
Jul 15 11:26:15.751698 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.751361    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" event={"ID":"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640","Type":"ContainerDied","Data":"f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c"}
Jul 15 11:26:15.751698 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.751375    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"
Jul 15 11:26:15.751698 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.751408    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" event={"ID":"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640","Type":"ContainerDied","Data":"dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020"}
Jul 15 11:26:15.751698 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.751429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g" event={"ID":"7c6da9a2-6ee3-4d8a-9d49-d7ae39701640","Type":"ContainerDied","Data":"debd3e4a6108054fd78dfb2763c12388257c896330ead3fd24633f0e6825b35a"}
Jul 15 11:26:15.759048 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.759032    2566 scope.go:117] "RemoveContainer" containerID="3d020bcd033a1473476266ec31b480cf1ef22acaa51026219d008a4a28dd0d5f"
Jul 15 11:26:15.767244 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.767230    2566 scope.go:117] "RemoveContainer" containerID="2abfe23934e8c7ee61c4fa52460f56fbe8b1d2b4ca60d4ad3b38fa46bdcac540"
Jul 15 11:26:15.767480 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:26:15.767464    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2abfe23934e8c7ee61c4fa52460f56fbe8b1d2b4ca60d4ad3b38fa46bdcac540\": container with ID starting with 2abfe23934e8c7ee61c4fa52460f56fbe8b1d2b4ca60d4ad3b38fa46bdcac540 not found: ID does not exist" containerID="2abfe23934e8c7ee61c4fa52460f56fbe8b1d2b4ca60d4ad3b38fa46bdcac540"
Jul 15 11:26:15.767556 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.767488    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2abfe23934e8c7ee61c4fa52460f56fbe8b1d2b4ca60d4ad3b38fa46bdcac540"} err="failed to get container status \"2abfe23934e8c7ee61c4fa52460f56fbe8b1d2b4ca60d4ad3b38fa46bdcac540\": rpc error: code = NotFound desc = could not find container \"2abfe23934e8c7ee61c4fa52460f56fbe8b1d2b4ca60d4ad3b38fa46bdcac540\": container with ID starting with 2abfe23934e8c7ee61c4fa52460f56fbe8b1d2b4ca60d4ad3b38fa46bdcac540 not found: ID does not exist"
Jul 15 11:26:15.767556 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.767519    2566 scope.go:117] "RemoveContainer" containerID="3d020bcd033a1473476266ec31b480cf1ef22acaa51026219d008a4a28dd0d5f"
Jul 15 11:26:15.767752 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:26:15.767733    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d020bcd033a1473476266ec31b480cf1ef22acaa51026219d008a4a28dd0d5f\": container with ID starting with 3d020bcd033a1473476266ec31b480cf1ef22acaa51026219d008a4a28dd0d5f not found: ID does not exist" containerID="3d020bcd033a1473476266ec31b480cf1ef22acaa51026219d008a4a28dd0d5f"
Jul 15 11:26:15.767793 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.767761    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d020bcd033a1473476266ec31b480cf1ef22acaa51026219d008a4a28dd0d5f"} err="failed to get container status \"3d020bcd033a1473476266ec31b480cf1ef22acaa51026219d008a4a28dd0d5f\": rpc error: code = NotFound desc = could not find container \"3d020bcd033a1473476266ec31b480cf1ef22acaa51026219d008a4a28dd0d5f\": container with ID starting with 3d020bcd033a1473476266ec31b480cf1ef22acaa51026219d008a4a28dd0d5f not found: ID does not exist"
Jul 15 11:26:15.767793 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.767776    2566 scope.go:117] "RemoveContainer" containerID="f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c"
Jul 15 11:26:15.774978 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.774959    2566 scope.go:117] "RemoveContainer" containerID="6df5072bc6ce0a6360a3db46f4303b67c24749fe290185cf5224f5966b73a712"
Jul 15 11:26:15.780017 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.779996    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"]
Jul 15 11:26:15.781386 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.781368    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67d6bfd74f-krzmk"]
Jul 15 11:26:15.783291 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.783263    2566 scope.go:117] "RemoveContainer" containerID="dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020"
Jul 15 11:26:15.790489 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.790473    2566 scope.go:117] "RemoveContainer" containerID="f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c"
Jul 15 11:26:15.790773 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:26:15.790755    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c\": container with ID starting with f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c not found: ID does not exist" containerID="f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c"
Jul 15 11:26:15.790835 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.790780    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c"} err="failed to get container status \"f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c\": rpc error: code = NotFound desc = could not find container \"f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c\": container with ID starting with f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c not found: ID does not exist"
Jul 15 11:26:15.790835 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.790800    2566 scope.go:117] "RemoveContainer" containerID="6df5072bc6ce0a6360a3db46f4303b67c24749fe290185cf5224f5966b73a712"
Jul 15 11:26:15.791036 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:26:15.791018    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df5072bc6ce0a6360a3db46f4303b67c24749fe290185cf5224f5966b73a712\": container with ID starting with 6df5072bc6ce0a6360a3db46f4303b67c24749fe290185cf5224f5966b73a712 not found: ID does not exist" containerID="6df5072bc6ce0a6360a3db46f4303b67c24749fe290185cf5224f5966b73a712"
Jul 15 11:26:15.791104 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.791045    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df5072bc6ce0a6360a3db46f4303b67c24749fe290185cf5224f5966b73a712"} err="failed to get container status \"6df5072bc6ce0a6360a3db46f4303b67c24749fe290185cf5224f5966b73a712\": rpc error: code = NotFound desc = could not find container \"6df5072bc6ce0a6360a3db46f4303b67c24749fe290185cf5224f5966b73a712\": container with ID starting with 6df5072bc6ce0a6360a3db46f4303b67c24749fe290185cf5224f5966b73a712 not found: ID does not exist"
Jul 15 11:26:15.791104 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.791068    2566 scope.go:117] "RemoveContainer" containerID="dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020"
Jul 15 11:26:15.791262 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:26:15.791245    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020\": container with ID starting with dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020 not found: ID does not exist" containerID="dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020"
Jul 15 11:26:15.791307 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.791265    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020"} err="failed to get container status \"dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020\": rpc error: code = NotFound desc = could not find container \"dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020\": container with ID starting with dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020 not found: ID does not exist"
Jul 15 11:26:15.791307 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.791279    2566 scope.go:117] "RemoveContainer" containerID="f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c"
Jul 15 11:26:15.791447 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.791430    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c"} err="failed to get container status \"f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c\": rpc error: code = NotFound desc = could not find container \"f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c\": container with ID starting with f5f9ccbce9ccdc35d103c5d9ce27941a01211301a339f45eaf65eb92ccf73e6c not found: ID does not exist"
Jul 15 11:26:15.791531 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.791448    2566 scope.go:117] "RemoveContainer" containerID="6df5072bc6ce0a6360a3db46f4303b67c24749fe290185cf5224f5966b73a712"
Jul 15 11:26:15.791730 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.791712    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df5072bc6ce0a6360a3db46f4303b67c24749fe290185cf5224f5966b73a712"} err="failed to get container status \"6df5072bc6ce0a6360a3db46f4303b67c24749fe290185cf5224f5966b73a712\": rpc error: code = NotFound desc = could not find container \"6df5072bc6ce0a6360a3db46f4303b67c24749fe290185cf5224f5966b73a712\": container with ID starting with 6df5072bc6ce0a6360a3db46f4303b67c24749fe290185cf5224f5966b73a712 not found: ID does not exist"
Jul 15 11:26:15.791730 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.791731    2566 scope.go:117] "RemoveContainer" containerID="dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020"
Jul 15 11:26:15.791965 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.791946    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020"} err="failed to get container status \"dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020\": rpc error: code = NotFound desc = could not find container \"dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020\": container with ID starting with dfab4c8294a502ce7967238e601c4993f87d83ac0e830c4c3d77c1b3a76ce020 not found: ID does not exist"
Jul 15 11:26:15.794262 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.794244    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"]
Jul 15 11:26:15.800849 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:15.800831    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-9f466b88c-2qt4g"]
Jul 15 11:26:17.267612 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:17.267580    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" path="/var/lib/kubelet/pods/7c6da9a2-6ee3-4d8a-9d49-d7ae39701640/volumes"
Jul 15 11:26:17.268041 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:17.268026    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" path="/var/lib/kubelet/pods/9e38acc6-5884-45a2-bb68-5b01b0c03279/volumes"
Jul 15 11:26:18.195511 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195466    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 15 11:26:18.195798 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195785    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="storage-initializer"
Jul 15 11:26:18.195850 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195800    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="storage-initializer"
Jul 15 11:26:18.195850 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195811    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main"
Jul 15 11:26:18.195850 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195817    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main"
Jul 15 11:26:18.195850 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195826    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="llm-d-routing-sidecar"
Jul 15 11:26:18.195850 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195832    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="llm-d-routing-sidecar"
Jul 15 11:26:18.195850 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195841    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main"
Jul 15 11:26:18.195850 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195848    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main"
Jul 15 11:26:18.196091 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195854    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main"
Jul 15 11:26:18.196091 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195859    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main"
Jul 15 11:26:18.196091 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195870    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main"
Jul 15 11:26:18.196091 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195876    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main"
Jul 15 11:26:18.196091 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195882    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="llm-d-routing-sidecar"
Jul 15 11:26:18.196091 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195886    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="llm-d-routing-sidecar"
Jul 15 11:26:18.196091 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195893    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="storage-initializer"
Jul 15 11:26:18.196091 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195898    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="storage-initializer"
Jul 15 11:26:18.196091 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195951    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="llm-d-routing-sidecar"
Jul 15 11:26:18.196091 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195957    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6da9a2-6ee3-4d8a-9d49-d7ae39701640" containerName="main"
Jul 15 11:26:18.196091 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195962    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="64b465b4-239d-477d-a664-0b5af7e88404" containerName="main"
Jul 15 11:26:18.196091 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195968    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="main"
Jul 15 11:26:18.196091 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195973    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d7f3079-4c4f-4edb-af19-2364c351eb7c" containerName="llm-d-routing-sidecar"
Jul 15 11:26:18.196091 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.195980    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e38acc6-5884-45a2-bb68-5b01b0c03279" containerName="main"
Jul 15 11:26:18.201165 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.201139    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.203740 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.203717    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 11:26:18.204604 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.204585    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-bv5qj\""
Jul 15 11:26:18.204705 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.204642    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 15 11:26:18.213251 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.213224    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 15 11:26:18.256706 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.256679    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 15 11:26:18.259866 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.259851    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.263675 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.263650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eef4ed1-ee90-479f-a819-2e6ea904da05-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.263778 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.263681    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp55v\" (UniqueName: \"kubernetes.io/projected/6eef4ed1-ee90-479f-a819-2e6ea904da05-kube-api-access-qp55v\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.263778 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.263699    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.263778 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.263745    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.263885 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.263802    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.263885 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.263828    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.273577 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.273554    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 15 11:26:18.364877 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.364845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.364877 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.364881    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxlqv\" (UniqueName: \"kubernetes.io/projected/188df47f-f5e6-4222-85bf-8f365481fe98-kube-api-access-cxlqv\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.365121 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.364924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.365121 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.364944    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188df47f-f5e6-4222-85bf-8f365481fe98-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.365121 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.364965    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eef4ed1-ee90-479f-a819-2e6ea904da05-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.365121 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.365018    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp55v\" (UniqueName: \"kubernetes.io/projected/6eef4ed1-ee90-479f-a819-2e6ea904da05-kube-api-access-qp55v\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.365121 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.365062    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.365121 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.365092    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.365388 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.365128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.365388 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.365155    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.365388 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.365254    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.365388 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.365280    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.365388 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.365318    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.365388 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.365346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.365633 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.365576    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.367450 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.367419    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.367670 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.367653    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eef4ed1-ee90-479f-a819-2e6ea904da05-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.375222 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.375193    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp55v\" (UniqueName: \"kubernetes.io/projected/6eef4ed1-ee90-479f-a819-2e6ea904da05-kube-api-access-qp55v\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:18.466408 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.466317    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.466408 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.466356    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188df47f-f5e6-4222-85bf-8f365481fe98-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.466408 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.466389    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.466719 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.466421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.466719 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.466472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.466719 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.466534    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxlqv\" (UniqueName: \"kubernetes.io/projected/188df47f-f5e6-4222-85bf-8f365481fe98-kube-api-access-cxlqv\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.466880 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.466761    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.466880 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.466788    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.466880 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.466827    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.468842 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.468814    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.469003 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.468986    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188df47f-f5e6-4222-85bf-8f365481fe98-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:18.475272 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:18.475251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxlqv\" (UniqueName: \"kubernetes.io/projected/188df47f-f5e6-4222-85bf-8f365481fe98-kube-api-access-cxlqv\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:20.787323 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:20.787292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:20.792961 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:20.792940    2566 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:20.888680 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:20.888641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:20.888839 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:20.888764    2566 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:20.891807 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:20.891789    2566 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") DevicePath \"csi-28b39141b605ecccebeeed3c7d0a1bb248dc5f88d802eb0dabea42669c8e3558\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:20.982966 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:20.982934    2566 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4767870384ba5a33f8c589260e567a3f76dd02f928fb126f91966e97c3d5e812/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:20.994288 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:20.994262    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:21.089895 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:21.089814    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:21.090051 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:21.089918    2566 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") DevicePath \"csi-28b39141b605ecccebeeed3c7d0a1bb248dc5f88d802eb0dabea42669c8e3558\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:21.092937 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:21.092921    2566 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") DevicePath \"csi-28b39141b605ecccebeeed3c7d0a1bb248dc5f88d802eb0dabea42669c8e3558\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:21.105083 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:21.105054    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:21.212456 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:21.212420    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:21.271738 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:21.270448    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:26:21.355799 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:21.355773    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 15 11:26:21.357952 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:26:21.357927    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eef4ed1_ee90_479f_a819_2e6ea904da05.slice/crio-2527099d5edaa9e40940ad4cdb858e9e3e7b3e72ec68af439f0298897256c8ce WatchSource:0}: Error finding container 2527099d5edaa9e40940ad4cdb858e9e3e7b3e72ec68af439f0298897256c8ce: Status 404 returned error can't find the container with id 2527099d5edaa9e40940ad4cdb858e9e3e7b3e72ec68af439f0298897256c8ce
Jul 15 11:26:21.416593 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:21.416565    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 15 11:26:21.419760 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:26:21.419721    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188df47f_f5e6_4222_85bf_8f365481fe98.slice/crio-85a0f8296967f44bd9000ac1d02b5dbad43daaa715bc19a41cfe69036c536aba WatchSource:0}: Error finding container 85a0f8296967f44bd9000ac1d02b5dbad43daaa715bc19a41cfe69036c536aba: Status 404 returned error can't find the container with id 85a0f8296967f44bd9000ac1d02b5dbad43daaa715bc19a41cfe69036c536aba
Jul 15 11:26:21.783439 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:21.783388    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"188df47f-f5e6-4222-85bf-8f365481fe98","Type":"ContainerStarted","Data":"12d521a3eff8b3dd207b1610e0a49b2286d6888318b3543bf9ea280b721425a4"}
Jul 15 11:26:21.783691 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:21.783457    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"188df47f-f5e6-4222-85bf-8f365481fe98","Type":"ContainerStarted","Data":"85a0f8296967f44bd9000ac1d02b5dbad43daaa715bc19a41cfe69036c536aba"}
Jul 15 11:26:21.785204 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:21.785162    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"6eef4ed1-ee90-479f-a819-2e6ea904da05","Type":"ContainerStarted","Data":"7dccf29e0602a04438a7b07644af6414b2007d921c90b9b614fb9a7154f15f5a"}
Jul 15 11:26:21.785204 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:21.785205    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"6eef4ed1-ee90-479f-a819-2e6ea904da05","Type":"ContainerStarted","Data":"2527099d5edaa9e40940ad4cdb858e9e3e7b3e72ec68af439f0298897256c8ce"}
Jul 15 11:26:21.805730 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:21.805659    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podStartSLOduration=3.805644417 podStartE2EDuration="3.805644417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:26:21.804066225 +0000 UTC m=+2451.034033761" watchObservedRunningTime="2026-07-15 11:26:21.805644417 +0000 UTC m=+2451.035611977"
Jul 15 11:26:21.828991 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:21.828941    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=3.828921769 podStartE2EDuration="3.828921769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:26:21.827885379 +0000 UTC m=+2451.057852912" watchObservedRunningTime="2026-07-15 11:26:21.828921769 +0000 UTC m=+2451.058889304"
Jul 15 11:26:22.212783 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:22.212687    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:31.213081 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:31.213041    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:26:31.214632 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:31.214596    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:26:34.558197 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.558162    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"]
Jul 15 11:26:34.561969 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.561940    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.565778 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.565753    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 15 11:26:34.565925 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.565794    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 11:26:34.566051 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.566024    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-rrn9p\""
Jul 15 11:26:34.575117 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.575091    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"]
Jul 15 11:26:34.579105 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.579083    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.579332 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.579185    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"]
Jul 15 11:26:34.593641 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.593613    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"]
Jul 15 11:26:34.619197 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.619167    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.619362 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.619310    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.619362 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.619345    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-dshm\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.619481 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.619376    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-model-cache\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.619481 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.619400    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd6tm\" (UniqueName: \"kubernetes.io/projected/0d939626-cfb4-49b0-92e0-3f193a6bf557-kube-api-access-bd6tm\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.619629 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.619528    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.619629 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.619613    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4292f99-135d-41ec-a124-ab786da15ea7-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.619736 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.619649    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d939626-cfb4-49b0-92e0-3f193a6bf557-tls-certs\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.619736 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.619693    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.619843 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.619812    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-home\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.619904 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.619852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.619960 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.619899    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-tmp-dir\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.619960 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.619927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-home\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.620055 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.619959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkhsw\" (UniqueName: \"kubernetes.io/projected/b4292f99-135d-41ec-a124-ab786da15ea7-kube-api-access-dkhsw\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.720866 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.720818    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.721080 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.720879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-dshm\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.721080 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.720912    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-model-cache\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.721080 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.720938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd6tm\" (UniqueName: \"kubernetes.io/projected/0d939626-cfb4-49b0-92e0-3f193a6bf557-kube-api-access-bd6tm\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.721080 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.720982    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.721414 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.721387    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-model-cache\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.721489 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.721414    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4292f99-135d-41ec-a124-ab786da15ea7-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.721566 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.721396    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.721620 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.721561    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d939626-cfb4-49b0-92e0-3f193a6bf557-tls-certs\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.721620 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.721592    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.721938 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.721914    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.722071 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.721590    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.722202 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.722124    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-home\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.722345 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.722306    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.722345 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.722366    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-tmp-dir\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.722549 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.722396    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-home\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.722549 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.722426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkhsw\" (UniqueName: \"kubernetes.io/projected/b4292f99-135d-41ec-a124-ab786da15ea7-kube-api-access-dkhsw\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.722549 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.722439    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-home\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.722549 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.722524    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.723131 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.723090    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-tmp-dir\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.723224 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.723143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-home\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.723289 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.723255    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.723880 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.723855    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-dshm\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.724405 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.724368    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4292f99-135d-41ec-a124-ab786da15ea7-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.724626 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.724602    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d939626-cfb4-49b0-92e0-3f193a6bf557-tls-certs\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.724971 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.724953    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.730126 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.730102    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd6tm\" (UniqueName: \"kubernetes.io/projected/0d939626-cfb4-49b0-92e0-3f193a6bf557-kube-api-access-bd6tm\") pod \"router-with-refs-pd-test-kserve-55fb5df567-zt9hf\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.731029 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.731005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkhsw\" (UniqueName: \"kubernetes.io/projected/b4292f99-135d-41ec-a124-ab786da15ea7-kube-api-access-dkhsw\") pod \"router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:34.876433 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.876337    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:34.892122 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:34.892094    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:35.045768 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:26:35.045734    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d939626_cfb4_49b0_92e0_3f193a6bf557.slice/crio-4ce96f98059b3c03bf942bfdc893e558aae38e51fe9a3eb2e7c4d8978f7234e4 WatchSource:0}: Error finding container 4ce96f98059b3c03bf942bfdc893e558aae38e51fe9a3eb2e7c4d8978f7234e4: Status 404 returned error can't find the container with id 4ce96f98059b3c03bf942bfdc893e558aae38e51fe9a3eb2e7c4d8978f7234e4
Jul 15 11:26:35.046772 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:35.046715    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"]
Jul 15 11:26:35.069002 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:35.068971    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"]
Jul 15 11:26:35.076701 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:26:35.076670    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4292f99_135d_41ec_a124_ab786da15ea7.slice/crio-743ad0982e94672139c43d57cb001f77ea1551cdd20050cb73ad043ca5509791 WatchSource:0}: Error finding container 743ad0982e94672139c43d57cb001f77ea1551cdd20050cb73ad043ca5509791: Status 404 returned error can't find the container with id 743ad0982e94672139c43d57cb001f77ea1551cdd20050cb73ad043ca5509791
Jul 15 11:26:35.859818 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:35.859778    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" event={"ID":"b4292f99-135d-41ec-a124-ab786da15ea7","Type":"ContainerStarted","Data":"c72568c45c1b49b9e39271c59419d4985389a08f6ac65275e523f691599bfaf3"}
Jul 15 11:26:35.860295 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:35.859844    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" event={"ID":"b4292f99-135d-41ec-a124-ab786da15ea7","Type":"ContainerStarted","Data":"743ad0982e94672139c43d57cb001f77ea1551cdd20050cb73ad043ca5509791"}
Jul 15 11:26:35.861873 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:35.861845    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" event={"ID":"0d939626-cfb4-49b0-92e0-3f193a6bf557","Type":"ContainerStarted","Data":"6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e"}
Jul 15 11:26:35.861996 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:35.861894    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" event={"ID":"0d939626-cfb4-49b0-92e0-3f193a6bf557","Type":"ContainerStarted","Data":"4ce96f98059b3c03bf942bfdc893e558aae38e51fe9a3eb2e7c4d8978f7234e4"}
Jul 15 11:26:35.861996 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:35.861923    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:36.873826 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:36.873786    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" event={"ID":"0d939626-cfb4-49b0-92e0-3f193a6bf557","Type":"ContainerStarted","Data":"5347870c7d792753b5be5e57a62195323fb1bd44384beb5879b4c4e4ad5c6ff0"}
Jul 15 11:26:38.903114 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:38.903079    2566 generic.go:356] "Generic (PLEG): container finished" podID="b4292f99-135d-41ec-a124-ab786da15ea7" containerID="c72568c45c1b49b9e39271c59419d4985389a08f6ac65275e523f691599bfaf3" exitCode=0
Jul 15 11:26:38.903745 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:38.903189    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" event={"ID":"b4292f99-135d-41ec-a124-ab786da15ea7","Type":"ContainerDied","Data":"c72568c45c1b49b9e39271c59419d4985389a08f6ac65275e523f691599bfaf3"}
Jul 15 11:26:39.919995 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:39.919946    2566 generic.go:356] "Generic (PLEG): container finished" podID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerID="5347870c7d792753b5be5e57a62195323fb1bd44384beb5879b4c4e4ad5c6ff0" exitCode=0
Jul 15 11:26:39.920474 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:39.920025    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" event={"ID":"0d939626-cfb4-49b0-92e0-3f193a6bf557","Type":"ContainerDied","Data":"5347870c7d792753b5be5e57a62195323fb1bd44384beb5879b4c4e4ad5c6ff0"}
Jul 15 11:26:39.922469 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:39.922444    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" event={"ID":"b4292f99-135d-41ec-a124-ab786da15ea7","Type":"ContainerStarted","Data":"5e70b81ab1b189fa6412d60951fd93754b789061d3c60223c7040c6016f613f3"}
Jul 15 11:26:39.972232 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:39.972145    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podStartSLOduration=2.697025309 podStartE2EDuration="5.972118055s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.275092746s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:26:39.969183391 +0000 UTC m=+2469.199150927" watchObservedRunningTime="2026-07-15 11:26:39.972118055 +0000 UTC m=+2469.202085588"
Jul 15 11:26:40.931956 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:40.931909    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" event={"ID":"0d939626-cfb4-49b0-92e0-3f193a6bf557","Type":"ContainerStarted","Data":"f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347"}
Jul 15 11:26:40.963240 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:40.963173    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podStartSLOduration=3.706081819 podStartE2EDuration="6.963147399s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.25706558s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:26:40.960765224 +0000 UTC m=+2470.190732761" watchObservedRunningTime="2026-07-15 11:26:40.963147399 +0000 UTC m=+2470.193114936"
Jul 15 11:26:41.213791 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:41.213674    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:26:44.877354 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:44.877296    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:44.877896 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:44.877433    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:26:44.879115 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:44.879077    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:26:44.893096 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:44.893041    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:44.893297 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:44.893134    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:26:44.894651 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:44.894617    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:26:51.213273 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:51.213217    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:26:54.877114 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:54.877062    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:26:54.893440 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:54.893394    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:26:54.899074 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:26:54.899038    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:27:01.213708 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:01.213647    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:27:04.877430 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:04.877372    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:27:04.893451 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:04.893397    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:27:11.213030 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:11.212980    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:27:14.877879 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:14.877826    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:27:14.893599 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:14.893550    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:27:21.213264 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:21.213216    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:27:24.876890 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:24.876827    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:27:24.894110 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:24.894066    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:27:31.213574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:31.213516    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:27:34.877980 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:34.877930    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:27:34.893020 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:34.892973    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:27:41.213407 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:41.213355    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:27:44.876876 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:44.876825    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:27:44.892828 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:44.892774    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:27:51.213035 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:51.212973    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:27:54.876984 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:54.876930    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:27:54.892643 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:27:54.892606    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:28:01.213204 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:01.213165    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:28:04.877151 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:04.877103    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:28:04.893473 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:04.893426    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:28:11.213217 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:11.213162    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:28:14.877657 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:14.877606    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:28:14.892694 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:14.892652    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:28:21.213748 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:21.213698    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:28:24.877514 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:24.877455    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:28:24.892655 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:24.892620    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:28:31.213081 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:31.213026    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:28:34.877798 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:34.877757    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:28:34.892864 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:34.892822    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:28:41.213123 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:41.213075    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:28:44.877667 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:44.877621    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:28:44.893490 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:44.893453    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:28:51.212824 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:51.212776    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:28:54.877729 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:54.877682    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:28:54.892792 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:28:54.892758    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:29:01.213792 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:29:01.213742    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:29:04.877595 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:29:04.877547    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:29:04.893133 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:29:04.893094    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:29:11.212951 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:29:11.212896    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:29:14.877643 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:29:14.877591    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:29:14.892864 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:29:14.892823    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:29:21.213031 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:29:21.212987    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:29:24.877722 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:29:24.877659    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:29:24.893383 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:29:24.893344    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:29:31.222683 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:29:31.222650    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:29:31.230272 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:29:31.230246    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:29:34.877682 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:29:34.877629    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:29:34.892396 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:29:34.892360    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:29:44.877081 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:29:44.877027    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:29:44.892551 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:29:44.892514    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:29:54.877238 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:29:54.877197    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Jul 15 11:29:54.892515 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:29:54.892464    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 15 11:30:04.886635 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:04.886599    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:30:04.898512 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:04.898474    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:30:04.902629 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:04.902606    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:30:04.910391 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:04.910367    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:30:14.297304 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:14.297270    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"]
Jul 15 11:30:14.303645 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:14.303620    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"]
Jul 15 11:30:29.321728 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:29.321633    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main" containerID="cri-o://5e70b81ab1b189fa6412d60951fd93754b789061d3c60223c7040c6016f613f3" gracePeriod=45
Jul 15 11:30:29.327161 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:29.327124    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main" containerID="cri-o://f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347" gracePeriod=45
Jul 15 11:30:30.145514 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.145459    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="llm-d-routing-sidecar" containerID="cri-o://6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e" gracePeriod=45
Jul 15 11:30:30.287990 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.287963    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:30:30.343645 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.343614    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-kserve-provision-location\") pod \"b4292f99-135d-41ec-a124-ab786da15ea7\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") "
Jul 15 11:30:30.344032 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.343658    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4292f99-135d-41ec-a124-ab786da15ea7-tls-certs\") pod \"b4292f99-135d-41ec-a124-ab786da15ea7\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") "
Jul 15 11:30:30.344032 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.343689    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-model-cache\") pod \"b4292f99-135d-41ec-a124-ab786da15ea7\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") "
Jul 15 11:30:30.344032 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.343708    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-tmp-dir\") pod \"b4292f99-135d-41ec-a124-ab786da15ea7\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") "
Jul 15 11:30:30.344032 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.343733    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-dshm\") pod \"b4292f99-135d-41ec-a124-ab786da15ea7\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") "
Jul 15 11:30:30.344032 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.343747    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-home\") pod \"b4292f99-135d-41ec-a124-ab786da15ea7\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") "
Jul 15 11:30:30.344032 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.343763    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkhsw\" (UniqueName: \"kubernetes.io/projected/b4292f99-135d-41ec-a124-ab786da15ea7-kube-api-access-dkhsw\") pod \"b4292f99-135d-41ec-a124-ab786da15ea7\" (UID: \"b4292f99-135d-41ec-a124-ab786da15ea7\") "
Jul 15 11:30:30.344361 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.344252    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4292f99-135d-41ec-a124-ab786da15ea7" (UID: "b4292f99-135d-41ec-a124-ab786da15ea7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:30:30.344652 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.344590    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-home" (OuterVolumeSpecName: "home") pod "b4292f99-135d-41ec-a124-ab786da15ea7" (UID: "b4292f99-135d-41ec-a124-ab786da15ea7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:30:30.346758 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.346713    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4292f99-135d-41ec-a124-ab786da15ea7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4292f99-135d-41ec-a124-ab786da15ea7" (UID: "b4292f99-135d-41ec-a124-ab786da15ea7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:30:30.346758 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.346719    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4292f99-135d-41ec-a124-ab786da15ea7-kube-api-access-dkhsw" (OuterVolumeSpecName: "kube-api-access-dkhsw") pod "b4292f99-135d-41ec-a124-ab786da15ea7" (UID: "b4292f99-135d-41ec-a124-ab786da15ea7"). InnerVolumeSpecName "kube-api-access-dkhsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:30:30.347162 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.347141    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-dshm" (OuterVolumeSpecName: "dshm") pod "b4292f99-135d-41ec-a124-ab786da15ea7" (UID: "b4292f99-135d-41ec-a124-ab786da15ea7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:30:30.365529 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.363180    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4292f99-135d-41ec-a124-ab786da15ea7" (UID: "b4292f99-135d-41ec-a124-ab786da15ea7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:30:30.403767 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.403710    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b4292f99-135d-41ec-a124-ab786da15ea7" (UID: "b4292f99-135d-41ec-a124-ab786da15ea7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:30:30.417751 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.417725    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:30:30.444376 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.444348    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-dshm\") pod \"0d939626-cfb4-49b0-92e0-3f193a6bf557\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") "
Jul 15 11:30:30.446124 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.444395    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-tmp-dir\") pod \"0d939626-cfb4-49b0-92e0-3f193a6bf557\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") "
Jul 15 11:30:30.446124 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.444424    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-home\") pod \"0d939626-cfb4-49b0-92e0-3f193a6bf557\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") "
Jul 15 11:30:30.446124 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.444452    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-kserve-provision-location\") pod \"0d939626-cfb4-49b0-92e0-3f193a6bf557\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") "
Jul 15 11:30:30.446124 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.444552    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-model-cache\") pod \"0d939626-cfb4-49b0-92e0-3f193a6bf557\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") "
Jul 15 11:30:30.446124 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.444596    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d939626-cfb4-49b0-92e0-3f193a6bf557-tls-certs\") pod \"0d939626-cfb4-49b0-92e0-3f193a6bf557\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") "
Jul 15 11:30:30.446124 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.444663    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd6tm\" (UniqueName: \"kubernetes.io/projected/0d939626-cfb4-49b0-92e0-3f193a6bf557-kube-api-access-bd6tm\") pod \"0d939626-cfb4-49b0-92e0-3f193a6bf557\" (UID: \"0d939626-cfb4-49b0-92e0-3f193a6bf557\") "
Jul 15 11:30:30.446124 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.444896    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d939626-cfb4-49b0-92e0-3f193a6bf557" (UID: "0d939626-cfb4-49b0-92e0-3f193a6bf557"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:30:30.446124 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.444963    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:30:30.446124 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.444983    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:30:30.446124 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.444999    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkhsw\" (UniqueName: \"kubernetes.io/projected/b4292f99-135d-41ec-a124-ab786da15ea7-kube-api-access-dkhsw\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:30:30.446124 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.445035    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:30:30.446124 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.445060    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4292f99-135d-41ec-a124-ab786da15ea7-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:30:30.446124 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.445074    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:30:30.446124 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.445087    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4292f99-135d-41ec-a124-ab786da15ea7-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:30:30.446124 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.445195    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-home" (OuterVolumeSpecName: "home") pod "0d939626-cfb4-49b0-92e0-3f193a6bf557" (UID: "0d939626-cfb4-49b0-92e0-3f193a6bf557"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:30:30.448310 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.448283    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d939626-cfb4-49b0-92e0-3f193a6bf557-kube-api-access-bd6tm" (OuterVolumeSpecName: "kube-api-access-bd6tm") pod "0d939626-cfb4-49b0-92e0-3f193a6bf557" (UID: "0d939626-cfb4-49b0-92e0-3f193a6bf557"). InnerVolumeSpecName "kube-api-access-bd6tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:30:30.449195 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.449171    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d939626-cfb4-49b0-92e0-3f193a6bf557-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d939626-cfb4-49b0-92e0-3f193a6bf557" (UID: "0d939626-cfb4-49b0-92e0-3f193a6bf557"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:30:30.449414 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.449395    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-dshm" (OuterVolumeSpecName: "dshm") pod "0d939626-cfb4-49b0-92e0-3f193a6bf557" (UID: "0d939626-cfb4-49b0-92e0-3f193a6bf557"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:30:30.464487 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.464455    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d939626-cfb4-49b0-92e0-3f193a6bf557" (UID: "0d939626-cfb4-49b0-92e0-3f193a6bf557"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:30:30.511573 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.511539    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d939626-cfb4-49b0-92e0-3f193a6bf557" (UID: "0d939626-cfb4-49b0-92e0-3f193a6bf557"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:30:30.545569 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.545539    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:30:30.545569 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.545563    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:30:30.545569 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.545571    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:30:30.545783 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.545579    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:30:30.545783 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.545588    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d939626-cfb4-49b0-92e0-3f193a6bf557-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:30:30.545783 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.545600    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d939626-cfb4-49b0-92e0-3f193a6bf557-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:30:30.545783 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:30.545608    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd6tm\" (UniqueName: \"kubernetes.io/projected/0d939626-cfb4-49b0-92e0-3f193a6bf557-kube-api-access-bd6tm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:30:31.136038 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.136002    2566 generic.go:356] "Generic (PLEG): container finished" podID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerID="f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347" exitCode=0
Jul 15 11:30:31.136038 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.136031    2566 generic.go:356] "Generic (PLEG): container finished" podID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerID="6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e" exitCode=0
Jul 15 11:30:31.136285 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.136098    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"
Jul 15 11:30:31.136285 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.136095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" event={"ID":"0d939626-cfb4-49b0-92e0-3f193a6bf557","Type":"ContainerDied","Data":"f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347"}
Jul 15 11:30:31.136285 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.136268    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" event={"ID":"0d939626-cfb4-49b0-92e0-3f193a6bf557","Type":"ContainerDied","Data":"6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e"}
Jul 15 11:30:31.136451 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.136296    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf" event={"ID":"0d939626-cfb4-49b0-92e0-3f193a6bf557","Type":"ContainerDied","Data":"4ce96f98059b3c03bf942bfdc893e558aae38e51fe9a3eb2e7c4d8978f7234e4"}
Jul 15 11:30:31.136451 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.136318    2566 scope.go:117] "RemoveContainer" containerID="f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347"
Jul 15 11:30:31.138427 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.138403    2566 generic.go:356] "Generic (PLEG): container finished" podID="b4292f99-135d-41ec-a124-ab786da15ea7" containerID="5e70b81ab1b189fa6412d60951fd93754b789061d3c60223c7040c6016f613f3" exitCode=0
Jul 15 11:30:31.138589 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.138437    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" event={"ID":"b4292f99-135d-41ec-a124-ab786da15ea7","Type":"ContainerDied","Data":"5e70b81ab1b189fa6412d60951fd93754b789061d3c60223c7040c6016f613f3"}
Jul 15 11:30:31.138589 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.138476    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k" event={"ID":"b4292f99-135d-41ec-a124-ab786da15ea7","Type":"ContainerDied","Data":"743ad0982e94672139c43d57cb001f77ea1551cdd20050cb73ad043ca5509791"}
Jul 15 11:30:31.138589 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.138481    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"
Jul 15 11:30:31.147385 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.147352    2566 scope.go:117] "RemoveContainer" containerID="5347870c7d792753b5be5e57a62195323fb1bd44384beb5879b4c4e4ad5c6ff0"
Jul 15 11:30:31.158577 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.158552    2566 scope.go:117] "RemoveContainer" containerID="6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e"
Jul 15 11:30:31.166125 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.166105    2566 scope.go:117] "RemoveContainer" containerID="f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347"
Jul 15 11:30:31.166319 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.166300    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"]
Jul 15 11:30:31.166398 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:30:31.166365    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347\": container with ID starting with f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347 not found: ID does not exist" containerID="f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347"
Jul 15 11:30:31.166459 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.166395    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347"} err="failed to get container status \"f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347\": rpc error: code = NotFound desc = could not find container \"f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347\": container with ID starting with f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347 not found: ID does not exist"
Jul 15 11:30:31.166459 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.166419    2566 scope.go:117] "RemoveContainer" containerID="5347870c7d792753b5be5e57a62195323fb1bd44384beb5879b4c4e4ad5c6ff0"
Jul 15 11:30:31.166718 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:30:31.166702    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5347870c7d792753b5be5e57a62195323fb1bd44384beb5879b4c4e4ad5c6ff0\": container with ID starting with 5347870c7d792753b5be5e57a62195323fb1bd44384beb5879b4c4e4ad5c6ff0 not found: ID does not exist" containerID="5347870c7d792753b5be5e57a62195323fb1bd44384beb5879b4c4e4ad5c6ff0"
Jul 15 11:30:31.166770 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.166734    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5347870c7d792753b5be5e57a62195323fb1bd44384beb5879b4c4e4ad5c6ff0"} err="failed to get container status \"5347870c7d792753b5be5e57a62195323fb1bd44384beb5879b4c4e4ad5c6ff0\": rpc error: code = NotFound desc = could not find container \"5347870c7d792753b5be5e57a62195323fb1bd44384beb5879b4c4e4ad5c6ff0\": container with ID starting with 5347870c7d792753b5be5e57a62195323fb1bd44384beb5879b4c4e4ad5c6ff0 not found: ID does not exist"
Jul 15 11:30:31.166770 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.166750    2566 scope.go:117] "RemoveContainer" containerID="6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e"
Jul 15 11:30:31.166976 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:30:31.166959    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e\": container with ID starting with 6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e not found: ID does not exist" containerID="6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e"
Jul 15 11:30:31.167021 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.166980    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e"} err="failed to get container status \"6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e\": rpc error: code = NotFound desc = could not find container \"6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e\": container with ID starting with 6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e not found: ID does not exist"
Jul 15 11:30:31.167021 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.166996    2566 scope.go:117] "RemoveContainer" containerID="f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347"
Jul 15 11:30:31.167183 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.167164    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347"} err="failed to get container status \"f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347\": rpc error: code = NotFound desc = could not find container \"f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347\": container with ID starting with f264ebaca54ec41f6d3f7ac210bad170e475e7bfd7b31aadb0d16f6eb7fa5347 not found: ID does not exist"
Jul 15 11:30:31.167247 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.167186    2566 scope.go:117] "RemoveContainer" containerID="5347870c7d792753b5be5e57a62195323fb1bd44384beb5879b4c4e4ad5c6ff0"
Jul 15 11:30:31.167438 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.167421    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5347870c7d792753b5be5e57a62195323fb1bd44384beb5879b4c4e4ad5c6ff0"} err="failed to get container status \"5347870c7d792753b5be5e57a62195323fb1bd44384beb5879b4c4e4ad5c6ff0\": rpc error: code = NotFound desc = could not find container \"5347870c7d792753b5be5e57a62195323fb1bd44384beb5879b4c4e4ad5c6ff0\": container with ID starting with 5347870c7d792753b5be5e57a62195323fb1bd44384beb5879b4c4e4ad5c6ff0 not found: ID does not exist"
Jul 15 11:30:31.167515 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.167439    2566 scope.go:117] "RemoveContainer" containerID="6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e"
Jul 15 11:30:31.167652 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.167628    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e"} err="failed to get container status \"6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e\": rpc error: code = NotFound desc = could not find container \"6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e\": container with ID starting with 6e3ed3be7fdeb94e93bbfbb1498b281fad1957bd9e47bea599879a1b2412198e not found: ID does not exist"
Jul 15 11:30:31.167695 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.167653    2566 scope.go:117] "RemoveContainer" containerID="5e70b81ab1b189fa6412d60951fd93754b789061d3c60223c7040c6016f613f3"
Jul 15 11:30:31.175808 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.175792    2566 scope.go:117] "RemoveContainer" containerID="c72568c45c1b49b9e39271c59419d4985389a08f6ac65275e523f691599bfaf3"
Jul 15 11:30:31.176535 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.176487    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55fb5df567-zt9hf"]
Jul 15 11:30:31.183421 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.183406    2566 scope.go:117] "RemoveContainer" containerID="5e70b81ab1b189fa6412d60951fd93754b789061d3c60223c7040c6016f613f3"
Jul 15 11:30:31.183698 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:30:31.183680    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e70b81ab1b189fa6412d60951fd93754b789061d3c60223c7040c6016f613f3\": container with ID starting with 5e70b81ab1b189fa6412d60951fd93754b789061d3c60223c7040c6016f613f3 not found: ID does not exist" containerID="5e70b81ab1b189fa6412d60951fd93754b789061d3c60223c7040c6016f613f3"
Jul 15 11:30:31.183748 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.183704    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e70b81ab1b189fa6412d60951fd93754b789061d3c60223c7040c6016f613f3"} err="failed to get container status \"5e70b81ab1b189fa6412d60951fd93754b789061d3c60223c7040c6016f613f3\": rpc error: code = NotFound desc = could not find container \"5e70b81ab1b189fa6412d60951fd93754b789061d3c60223c7040c6016f613f3\": container with ID starting with 5e70b81ab1b189fa6412d60951fd93754b789061d3c60223c7040c6016f613f3 not found: ID does not exist"
Jul 15 11:30:31.183748 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.183719    2566 scope.go:117] "RemoveContainer" containerID="c72568c45c1b49b9e39271c59419d4985389a08f6ac65275e523f691599bfaf3"
Jul 15 11:30:31.183963 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:30:31.183948    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72568c45c1b49b9e39271c59419d4985389a08f6ac65275e523f691599bfaf3\": container with ID starting with c72568c45c1b49b9e39271c59419d4985389a08f6ac65275e523f691599bfaf3 not found: ID does not exist" containerID="c72568c45c1b49b9e39271c59419d4985389a08f6ac65275e523f691599bfaf3"
Jul 15 11:30:31.184016 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.183965    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72568c45c1b49b9e39271c59419d4985389a08f6ac65275e523f691599bfaf3"} err="failed to get container status \"c72568c45c1b49b9e39271c59419d4985389a08f6ac65275e523f691599bfaf3\": rpc error: code = NotFound desc = could not find container \"c72568c45c1b49b9e39271c59419d4985389a08f6ac65275e523f691599bfaf3\": container with ID starting with c72568c45c1b49b9e39271c59419d4985389a08f6ac65275e523f691599bfaf3 not found: ID does not exist"
Jul 15 11:30:31.186720 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.186700    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"]
Jul 15 11:30:31.195818 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.195793    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6bd9dbd5d5-qnz7k"]
Jul 15 11:30:31.268641 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.268614    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" path="/var/lib/kubelet/pods/0d939626-cfb4-49b0-92e0-3f193a6bf557/volumes"
Jul 15 11:30:31.269092 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:31.269078    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" path="/var/lib/kubelet/pods/b4292f99-135d-41ec-a124-ab786da15ea7/volumes"
Jul 15 11:30:38.400192 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:38.400161    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:30:38.400691 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:38.400161    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:30:40.326788 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.326754    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"]
Jul 15 11:30:40.327235 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.327218    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main"
Jul 15 11:30:40.327283 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.327239    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main"
Jul 15 11:30:40.327283 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.327258    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="storage-initializer"
Jul 15 11:30:40.327283 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.327267    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="storage-initializer"
Jul 15 11:30:40.327283 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.327279    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="storage-initializer"
Jul 15 11:30:40.327415 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.327288    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="storage-initializer"
Jul 15 11:30:40.327415 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.327317    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main"
Jul 15 11:30:40.327415 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.327326    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main"
Jul 15 11:30:40.327415 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.327338    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="llm-d-routing-sidecar"
Jul 15 11:30:40.327415 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.327345    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="llm-d-routing-sidecar"
Jul 15 11:30:40.327415 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.327406    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4292f99-135d-41ec-a124-ab786da15ea7" containerName="main"
Jul 15 11:30:40.327621 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.327418    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="main"
Jul 15 11:30:40.327621 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.327430    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d939626-cfb4-49b0-92e0-3f193a6bf557" containerName="llm-d-routing-sidecar"
Jul 15 11:30:40.330565 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.330548    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.335886 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.335861    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 11:30:40.336035 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.336015    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 15 11:30:40.344239 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.344216    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"]
Jul 15 11:30:40.435276 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.435239    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.435276 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.435281    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.435538 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.435300    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sckct\" (UniqueName: \"kubernetes.io/projected/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-kube-api-access-sckct\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.435538 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.435403    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.435538 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.435437    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.435538 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.435481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.435685 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.435565    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.536488 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.536454    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.536735 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.536549    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.536735 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.536591    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.536735 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.536627    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.536735 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.536651    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sckct\" (UniqueName: \"kubernetes.io/projected/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-kube-api-access-sckct\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.536735 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.536704    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.536735 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.536735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.537077 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.536924    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.537077 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.537016    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.537077 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.537071    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.537190 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.537133    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.539053 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.539029    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.539299 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.539280    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.545673 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.545651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sckct\" (UniqueName: \"kubernetes.io/projected/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-kube-api-access-sckct\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.640354 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.640264    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:40.781020 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:40.780986    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"]
Jul 15 11:30:40.784003 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:30:40.783973    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3863c3e6_fd69_4381_ba9d_9648ae4cd11f.slice/crio-be1911ebb99fde895ab58c369227eb16f73613bdf663bd86943cffe9f82f3758 WatchSource:0}: Error finding container be1911ebb99fde895ab58c369227eb16f73613bdf663bd86943cffe9f82f3758: Status 404 returned error can't find the container with id be1911ebb99fde895ab58c369227eb16f73613bdf663bd86943cffe9f82f3758
Jul 15 11:30:41.193563 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:41.193529    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" event={"ID":"3863c3e6-fd69-4381-ba9d-9648ae4cd11f","Type":"ContainerStarted","Data":"678f38694e787229b53e3261e89178c394a3038bcd26bca77d6b40746db61a64"}
Jul 15 11:30:41.193746 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:41.193602    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" event={"ID":"3863c3e6-fd69-4381-ba9d-9648ae4cd11f","Type":"ContainerStarted","Data":"be1911ebb99fde895ab58c369227eb16f73613bdf663bd86943cffe9f82f3758"}
Jul 15 11:30:44.208092 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:44.208007    2566 generic.go:356] "Generic (PLEG): container finished" podID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerID="678f38694e787229b53e3261e89178c394a3038bcd26bca77d6b40746db61a64" exitCode=0
Jul 15 11:30:44.208092 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:44.208061    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" event={"ID":"3863c3e6-fd69-4381-ba9d-9648ae4cd11f","Type":"ContainerDied","Data":"678f38694e787229b53e3261e89178c394a3038bcd26bca77d6b40746db61a64"}
Jul 15 11:30:45.214475 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:45.214445    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" event={"ID":"3863c3e6-fd69-4381-ba9d-9648ae4cd11f","Type":"ContainerStarted","Data":"6dc93b64bca242eaa03989d917dbcbb7c281c223dd49af6f0a01d5533ae15a5b"}
Jul 15 11:30:45.242757 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:45.242706    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" podStartSLOduration=2.284149068 podStartE2EDuration="5.242690494s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.958541426s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:30:45.240030877 +0000 UTC m=+2714.469998407" watchObservedRunningTime="2026-07-15 11:30:45.242690494 +0000 UTC m=+2714.472658072"
Jul 15 11:30:45.641141 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:45.641104    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:50.640632 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:50.640593    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:30:50.642197 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:30:50.642169    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 15 11:31:00.641511 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:31:00.641452    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 15 11:31:10.641391 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:31:10.641346    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 15 11:31:20.641022 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:31:20.640977    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 15 11:31:30.641388 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:31:30.641340    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 15 11:31:40.641284 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:31:40.641241    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 15 11:31:50.641206 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:31:50.641152    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 15 11:32:00.641129 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:32:00.641021    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 15 11:32:10.640759 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:32:10.640710    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 15 11:32:20.640871 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:32:20.640828    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 15 11:32:30.641511 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:32:30.641458    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 15 11:32:40.651541 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:32:40.651490    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:32:40.659394 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:32:40.659369    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:32:45.618235 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:32:45.618198    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"]
Jul 15 11:33:00.641896 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:00.641836    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerName="main" containerID="cri-o://6dc93b64bca242eaa03989d917dbcbb7c281c223dd49af6f0a01d5533ae15a5b" gracePeriod=45
Jul 15 11:33:01.309859 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.309830    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:33:01.376196 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.376162    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-home\") pod \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") "
Jul 15 11:33:01.376373 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.376224    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-kserve-provision-location\") pod \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") "
Jul 15 11:33:01.376373 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.376268    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sckct\" (UniqueName: \"kubernetes.io/projected/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-kube-api-access-sckct\") pod \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") "
Jul 15 11:33:01.376373 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.376285    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-tmp-dir\") pod \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") "
Jul 15 11:33:01.376373 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.376300    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-model-cache\") pod \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") "
Jul 15 11:33:01.376373 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.376341    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-dshm\") pod \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") "
Jul 15 11:33:01.376674 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.376368    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-tls-certs\") pod \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\" (UID: \"3863c3e6-fd69-4381-ba9d-9648ae4cd11f\") "
Jul 15 11:33:01.376736 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.376692    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-model-cache" (OuterVolumeSpecName: "model-cache") pod "3863c3e6-fd69-4381-ba9d-9648ae4cd11f" (UID: "3863c3e6-fd69-4381-ba9d-9648ae4cd11f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:33:01.377045 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.377014    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:33:01.377544 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.377512    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-home" (OuterVolumeSpecName: "home") pod "3863c3e6-fd69-4381-ba9d-9648ae4cd11f" (UID: "3863c3e6-fd69-4381-ba9d-9648ae4cd11f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:33:01.378786 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.378756    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-dshm" (OuterVolumeSpecName: "dshm") pod "3863c3e6-fd69-4381-ba9d-9648ae4cd11f" (UID: "3863c3e6-fd69-4381-ba9d-9648ae4cd11f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:33:01.378881 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.378815    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3863c3e6-fd69-4381-ba9d-9648ae4cd11f" (UID: "3863c3e6-fd69-4381-ba9d-9648ae4cd11f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:33:01.378978 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.378963    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-kube-api-access-sckct" (OuterVolumeSpecName: "kube-api-access-sckct") pod "3863c3e6-fd69-4381-ba9d-9648ae4cd11f" (UID: "3863c3e6-fd69-4381-ba9d-9648ae4cd11f"). InnerVolumeSpecName "kube-api-access-sckct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:33:01.394404 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.394372    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3863c3e6-fd69-4381-ba9d-9648ae4cd11f" (UID: "3863c3e6-fd69-4381-ba9d-9648ae4cd11f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:33:01.442510 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.442398    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3863c3e6-fd69-4381-ba9d-9648ae4cd11f" (UID: "3863c3e6-fd69-4381-ba9d-9648ae4cd11f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:33:01.477737 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.477702    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:33:01.477737 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.477734    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:33:01.477913 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.477746    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sckct\" (UniqueName: \"kubernetes.io/projected/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-kube-api-access-sckct\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:33:01.477913 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.477755    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:33:01.477913 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.477764    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:33:01.477913 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.477772    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3863c3e6-fd69-4381-ba9d-9648ae4cd11f-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:33:01.852740 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.852702    2566 generic.go:356] "Generic (PLEG): container finished" podID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerID="6dc93b64bca242eaa03989d917dbcbb7c281c223dd49af6f0a01d5533ae15a5b" exitCode=0
Jul 15 11:33:01.853165 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.852755    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"
Jul 15 11:33:01.853165 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.852783    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" event={"ID":"3863c3e6-fd69-4381-ba9d-9648ae4cd11f","Type":"ContainerDied","Data":"6dc93b64bca242eaa03989d917dbcbb7c281c223dd49af6f0a01d5533ae15a5b"}
Jul 15 11:33:01.853165 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.852838    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq" event={"ID":"3863c3e6-fd69-4381-ba9d-9648ae4cd11f","Type":"ContainerDied","Data":"be1911ebb99fde895ab58c369227eb16f73613bdf663bd86943cffe9f82f3758"}
Jul 15 11:33:01.853165 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.852861    2566 scope.go:117] "RemoveContainer" containerID="6dc93b64bca242eaa03989d917dbcbb7c281c223dd49af6f0a01d5533ae15a5b"
Jul 15 11:33:01.862059 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.862042    2566 scope.go:117] "RemoveContainer" containerID="678f38694e787229b53e3261e89178c394a3038bcd26bca77d6b40746db61a64"
Jul 15 11:33:01.870278 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.870259    2566 scope.go:117] "RemoveContainer" containerID="6dc93b64bca242eaa03989d917dbcbb7c281c223dd49af6f0a01d5533ae15a5b"
Jul 15 11:33:01.870582 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:33:01.870550    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc93b64bca242eaa03989d917dbcbb7c281c223dd49af6f0a01d5533ae15a5b\": container with ID starting with 6dc93b64bca242eaa03989d917dbcbb7c281c223dd49af6f0a01d5533ae15a5b not found: ID does not exist" containerID="6dc93b64bca242eaa03989d917dbcbb7c281c223dd49af6f0a01d5533ae15a5b"
Jul 15 11:33:01.870683 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.870588    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc93b64bca242eaa03989d917dbcbb7c281c223dd49af6f0a01d5533ae15a5b"} err="failed to get container status \"6dc93b64bca242eaa03989d917dbcbb7c281c223dd49af6f0a01d5533ae15a5b\": rpc error: code = NotFound desc = could not find container \"6dc93b64bca242eaa03989d917dbcbb7c281c223dd49af6f0a01d5533ae15a5b\": container with ID starting with 6dc93b64bca242eaa03989d917dbcbb7c281c223dd49af6f0a01d5533ae15a5b not found: ID does not exist"
Jul 15 11:33:01.870683 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.870613    2566 scope.go:117] "RemoveContainer" containerID="678f38694e787229b53e3261e89178c394a3038bcd26bca77d6b40746db61a64"
Jul 15 11:33:01.870860 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:33:01.870843    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"678f38694e787229b53e3261e89178c394a3038bcd26bca77d6b40746db61a64\": container with ID starting with 678f38694e787229b53e3261e89178c394a3038bcd26bca77d6b40746db61a64 not found: ID does not exist" containerID="678f38694e787229b53e3261e89178c394a3038bcd26bca77d6b40746db61a64"
Jul 15 11:33:01.870906 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.870866    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678f38694e787229b53e3261e89178c394a3038bcd26bca77d6b40746db61a64"} err="failed to get container status \"678f38694e787229b53e3261e89178c394a3038bcd26bca77d6b40746db61a64\": rpc error: code = NotFound desc = could not find container \"678f38694e787229b53e3261e89178c394a3038bcd26bca77d6b40746db61a64\": container with ID starting with 678f38694e787229b53e3261e89178c394a3038bcd26bca77d6b40746db61a64 not found: ID does not exist"
Jul 15 11:33:01.875344 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.875321    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"]
Jul 15 11:33:01.879569 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:01.879546    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-677888c56b28djq"]
Jul 15 11:33:03.267655 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:03.267621    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" path="/var/lib/kubelet/pods/3863c3e6-fd69-4381-ba9d-9648ae4cd11f/volumes"
Jul 15 11:33:18.560711 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.560677    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 15 11:33:18.561135 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.561004    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerName="storage-initializer"
Jul 15 11:33:18.561135 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.561017    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerName="storage-initializer"
Jul 15 11:33:18.561135 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.561029    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerName="main"
Jul 15 11:33:18.561135 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.561034    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerName="main"
Jul 15 11:33:18.561135 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.561104    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3863c3e6-fd69-4381-ba9d-9648ae4cd11f" containerName="main"
Jul 15 11:33:18.563067 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.563050    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.565672 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.565652    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 11:33:18.565798 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.565740    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 15 11:33:18.566427 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.566411    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-l8jnh\""
Jul 15 11:33:18.578567 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.578545    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 15 11:33:18.636427 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.636398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.636605 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.636451    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzr7f\" (UniqueName: \"kubernetes.io/projected/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-kube-api-access-zzr7f\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.636605 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.636478    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.636605 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.636527    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.636605 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.636552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.636782 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.636621    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.636782 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.636665    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.737131 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.737094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.737300 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.737158    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.737300 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.737207    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzr7f\" (UniqueName: \"kubernetes.io/projected/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-kube-api-access-zzr7f\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.737300 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.737233    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.737300 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.737263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.737300 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.737290    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.737644 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.737334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.737644 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.737482    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.737644 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.737545    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.737754 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.737676    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.737754 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.737712    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.739662 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.739640    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.739807 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.739793    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.750140 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.750111    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzr7f\" (UniqueName: \"kubernetes.io/projected/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-kube-api-access-zzr7f\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:18.873284 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:18.873198    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:19.019865 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:19.019836    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 15 11:33:19.023066 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:33:19.023040    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1508ae0_d9cb_4c7a_83ed_4a73028d5196.slice/crio-8dc181b81df0dd059931ea6c54728a2963f043858a36fae0e3ab3e71d3757f73 WatchSource:0}: Error finding container 8dc181b81df0dd059931ea6c54728a2963f043858a36fae0e3ab3e71d3757f73: Status 404 returned error can't find the container with id 8dc181b81df0dd059931ea6c54728a2963f043858a36fae0e3ab3e71d3757f73
Jul 15 11:33:19.940167 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:19.940115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e1508ae0-d9cb-4c7a-83ed-4a73028d5196","Type":"ContainerStarted","Data":"e34a9fe5c4774842c35563ec9748af925192744af4737afeee7cd35f2195a7b4"}
Jul 15 11:33:19.940667 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:19.940186    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e1508ae0-d9cb-4c7a-83ed-4a73028d5196","Type":"ContainerStarted","Data":"8dc181b81df0dd059931ea6c54728a2963f043858a36fae0e3ab3e71d3757f73"}
Jul 15 11:33:22.956813 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:22.956777    2566 generic.go:356] "Generic (PLEG): container finished" podID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerID="e34a9fe5c4774842c35563ec9748af925192744af4737afeee7cd35f2195a7b4" exitCode=0
Jul 15 11:33:22.957216 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:22.956851    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e1508ae0-d9cb-4c7a-83ed-4a73028d5196","Type":"ContainerDied","Data":"e34a9fe5c4774842c35563ec9748af925192744af4737afeee7cd35f2195a7b4"}
Jul 15 11:33:23.964064 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:23.964029    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e1508ae0-d9cb-4c7a-83ed-4a73028d5196","Type":"ContainerStarted","Data":"43e55591f5238c44721ebd5d423903f17fee7940980f05504d69c544ae206c71"}
Jul 15 11:33:24.874076 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:24.874038    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:28.874194 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:28.874149    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:33:28.875828 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:28.875793    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 15 11:33:38.874669 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:38.874617    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 15 11:33:48.874144 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:48.874090    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 15 11:33:58.874332 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:58.874268    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 15 11:33:59.492442 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:59.492383    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=38.511789499 podStartE2EDuration="41.492365281s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.980575782s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:33:23.987635757 +0000 UTC m=+2873.217603290" watchObservedRunningTime="2026-07-15 11:33:59.492365281 +0000 UTC m=+2908.722332812"
Jul 15 11:33:59.493675 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:33:59.493651    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 15 11:34:08.874104 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:08.874058    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 15 11:34:14.524314 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:14.524254    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" containerID="cri-o://7dccf29e0602a04438a7b07644af6414b2007d921c90b9b614fb9a7154f15f5a" gracePeriod=45
Jul 15 11:34:15.223476 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.223424    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 15 11:34:15.504219 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.504195    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:34:15.682270 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.682232    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-home\") pod \"6eef4ed1-ee90-479f-a819-2e6ea904da05\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") "
Jul 15 11:34:15.682717 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.682382    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"6eef4ed1-ee90-479f-a819-2e6ea904da05\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") "
Jul 15 11:34:15.682717 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.682419    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-tmp-dir\") pod \"6eef4ed1-ee90-479f-a819-2e6ea904da05\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") "
Jul 15 11:34:15.682717 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.682467    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eef4ed1-ee90-479f-a819-2e6ea904da05-tls-certs\") pod \"6eef4ed1-ee90-479f-a819-2e6ea904da05\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") "
Jul 15 11:34:15.682717 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.682552    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-dshm\") pod \"6eef4ed1-ee90-479f-a819-2e6ea904da05\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") "
Jul 15 11:34:15.682717 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.682588    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp55v\" (UniqueName: \"kubernetes.io/projected/6eef4ed1-ee90-479f-a819-2e6ea904da05-kube-api-access-qp55v\") pod \"6eef4ed1-ee90-479f-a819-2e6ea904da05\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") "
Jul 15 11:34:15.682717 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.682613    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-model-cache\") pod \"6eef4ed1-ee90-479f-a819-2e6ea904da05\" (UID: \"6eef4ed1-ee90-479f-a819-2e6ea904da05\") "
Jul 15 11:34:15.683041 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.682951    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-home" (OuterVolumeSpecName: "home") pod "6eef4ed1-ee90-479f-a819-2e6ea904da05" (UID: "6eef4ed1-ee90-479f-a819-2e6ea904da05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:34:15.683041 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.683035    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-model-cache" (OuterVolumeSpecName: "model-cache") pod "6eef4ed1-ee90-479f-a819-2e6ea904da05" (UID: "6eef4ed1-ee90-479f-a819-2e6ea904da05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:34:15.685275 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.685238    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eef4ed1-ee90-479f-a819-2e6ea904da05-kube-api-access-qp55v" (OuterVolumeSpecName: "kube-api-access-qp55v") pod "6eef4ed1-ee90-479f-a819-2e6ea904da05" (UID: "6eef4ed1-ee90-479f-a819-2e6ea904da05"). InnerVolumeSpecName "kube-api-access-qp55v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:34:15.685424 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.685327    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eef4ed1-ee90-479f-a819-2e6ea904da05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6eef4ed1-ee90-479f-a819-2e6ea904da05" (UID: "6eef4ed1-ee90-479f-a819-2e6ea904da05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:34:15.685534 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.685483    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-dshm" (OuterVolumeSpecName: "dshm") pod "6eef4ed1-ee90-479f-a819-2e6ea904da05" (UID: "6eef4ed1-ee90-479f-a819-2e6ea904da05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:34:15.686035 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.686010    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4" (OuterVolumeSpecName: "kserve-pvc-source") pod "6eef4ed1-ee90-479f-a819-2e6ea904da05" (UID: "6eef4ed1-ee90-479f-a819-2e6ea904da05"). InnerVolumeSpecName "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 15 11:34:15.695276 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.695247    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6eef4ed1-ee90-479f-a819-2e6ea904da05" (UID: "6eef4ed1-ee90-479f-a819-2e6ea904da05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:34:15.784153 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.784119    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:34:15.784153 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.784147    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eef4ed1-ee90-479f-a819-2e6ea904da05-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:34:15.784153 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.784157    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:34:15.784376 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.784166    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp55v\" (UniqueName: \"kubernetes.io/projected/6eef4ed1-ee90-479f-a819-2e6ea904da05-kube-api-access-qp55v\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:34:15.784376 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.784174    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:34:15.784376 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:15.784181    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eef4ed1-ee90-479f-a819-2e6ea904da05-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:34:16.212931 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:16.212825    2566 generic.go:356] "Generic (PLEG): container finished" podID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerID="7dccf29e0602a04438a7b07644af6414b2007d921c90b9b614fb9a7154f15f5a" exitCode=0
Jul 15 11:34:16.212931 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:16.212902    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 15 11:34:16.213135 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:16.212898    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"6eef4ed1-ee90-479f-a819-2e6ea904da05","Type":"ContainerDied","Data":"7dccf29e0602a04438a7b07644af6414b2007d921c90b9b614fb9a7154f15f5a"}
Jul 15 11:34:16.213135 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:16.213070    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"6eef4ed1-ee90-479f-a819-2e6ea904da05","Type":"ContainerDied","Data":"2527099d5edaa9e40940ad4cdb858e9e3e7b3e72ec68af439f0298897256c8ce"}
Jul 15 11:34:16.213135 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:16.213094    2566 scope.go:117] "RemoveContainer" containerID="7dccf29e0602a04438a7b07644af6414b2007d921c90b9b614fb9a7154f15f5a"
Jul 15 11:34:16.222065 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:16.222040    2566 scope.go:117] "RemoveContainer" containerID="7dccf29e0602a04438a7b07644af6414b2007d921c90b9b614fb9a7154f15f5a"
Jul 15 11:34:16.222398 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:34:16.222368    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dccf29e0602a04438a7b07644af6414b2007d921c90b9b614fb9a7154f15f5a\": container with ID starting with 7dccf29e0602a04438a7b07644af6414b2007d921c90b9b614fb9a7154f15f5a not found: ID does not exist" containerID="7dccf29e0602a04438a7b07644af6414b2007d921c90b9b614fb9a7154f15f5a"
Jul 15 11:34:16.222490 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:16.222406    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dccf29e0602a04438a7b07644af6414b2007d921c90b9b614fb9a7154f15f5a"} err="failed to get container status \"7dccf29e0602a04438a7b07644af6414b2007d921c90b9b614fb9a7154f15f5a\": rpc error: code = NotFound desc = could not find container \"7dccf29e0602a04438a7b07644af6414b2007d921c90b9b614fb9a7154f15f5a\": container with ID starting with 7dccf29e0602a04438a7b07644af6414b2007d921c90b9b614fb9a7154f15f5a not found: ID does not exist"
Jul 15 11:34:16.236996 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:16.236972    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 15 11:34:16.243613 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:16.243587    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 15 11:34:16.260997 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:16.260963    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 15 11:34:17.267782 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:17.267748    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" path="/var/lib/kubelet/pods/6eef4ed1-ee90-479f-a819-2e6ea904da05/volumes"
Jul 15 11:34:18.874043 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:18.873991    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 15 11:34:28.874048 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:28.874001    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 15 11:34:31.284267 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:31.284228    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="188df47f-f5e6-4222-85bf-8f365481fe98" containerName="main" containerID="cri-o://12d521a3eff8b3dd207b1610e0a49b2286d6888318b3543bf9ea280b721425a4" gracePeriod=45
Jul 15 11:34:32.240312 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.240290    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:34:32.293384 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.293348    2566 generic.go:356] "Generic (PLEG): container finished" podID="188df47f-f5e6-4222-85bf-8f365481fe98" containerID="12d521a3eff8b3dd207b1610e0a49b2286d6888318b3543bf9ea280b721425a4" exitCode=0
Jul 15 11:34:32.293866 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.293391    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 15 11:34:32.293866 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.293417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"188df47f-f5e6-4222-85bf-8f365481fe98","Type":"ContainerDied","Data":"12d521a3eff8b3dd207b1610e0a49b2286d6888318b3543bf9ea280b721425a4"}
Jul 15 11:34:32.293866 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.293474    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"188df47f-f5e6-4222-85bf-8f365481fe98","Type":"ContainerDied","Data":"85a0f8296967f44bd9000ac1d02b5dbad43daaa715bc19a41cfe69036c536aba"}
Jul 15 11:34:32.293866 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.293530    2566 scope.go:117] "RemoveContainer" containerID="12d521a3eff8b3dd207b1610e0a49b2286d6888318b3543bf9ea280b721425a4"
Jul 15 11:34:32.302172 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.302153    2566 scope.go:117] "RemoveContainer" containerID="12d521a3eff8b3dd207b1610e0a49b2286d6888318b3543bf9ea280b721425a4"
Jul 15 11:34:32.302425 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:34:32.302406    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d521a3eff8b3dd207b1610e0a49b2286d6888318b3543bf9ea280b721425a4\": container with ID starting with 12d521a3eff8b3dd207b1610e0a49b2286d6888318b3543bf9ea280b721425a4 not found: ID does not exist" containerID="12d521a3eff8b3dd207b1610e0a49b2286d6888318b3543bf9ea280b721425a4"
Jul 15 11:34:32.302476 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.302439    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d521a3eff8b3dd207b1610e0a49b2286d6888318b3543bf9ea280b721425a4"} err="failed to get container status \"12d521a3eff8b3dd207b1610e0a49b2286d6888318b3543bf9ea280b721425a4\": rpc error: code = NotFound desc = could not find container \"12d521a3eff8b3dd207b1610e0a49b2286d6888318b3543bf9ea280b721425a4\": container with ID starting with 12d521a3eff8b3dd207b1610e0a49b2286d6888318b3543bf9ea280b721425a4 not found: ID does not exist"
Jul 15 11:34:32.326949 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.326889    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxlqv\" (UniqueName: \"kubernetes.io/projected/188df47f-f5e6-4222-85bf-8f365481fe98-kube-api-access-cxlqv\") pod \"188df47f-f5e6-4222-85bf-8f365481fe98\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") "
Jul 15 11:34:32.326949 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.326927    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188df47f-f5e6-4222-85bf-8f365481fe98-tls-certs\") pod \"188df47f-f5e6-4222-85bf-8f365481fe98\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") "
Jul 15 11:34:32.326949 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.326945    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-tmp-dir\") pod \"188df47f-f5e6-4222-85bf-8f365481fe98\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") "
Jul 15 11:34:32.327171 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.326986    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-dshm\") pod \"188df47f-f5e6-4222-85bf-8f365481fe98\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") "
Jul 15 11:34:32.327171 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.327029    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-model-cache\") pod \"188df47f-f5e6-4222-85bf-8f365481fe98\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") "
Jul 15 11:34:32.327171 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.327102    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") pod \"188df47f-f5e6-4222-85bf-8f365481fe98\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") "
Jul 15 11:34:32.327171 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.327162    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-home\") pod \"188df47f-f5e6-4222-85bf-8f365481fe98\" (UID: \"188df47f-f5e6-4222-85bf-8f365481fe98\") "
Jul 15 11:34:32.327548 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.327476    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-model-cache" (OuterVolumeSpecName: "model-cache") pod "188df47f-f5e6-4222-85bf-8f365481fe98" (UID: "188df47f-f5e6-4222-85bf-8f365481fe98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:34:32.327875 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.327847    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-home" (OuterVolumeSpecName: "home") pod "188df47f-f5e6-4222-85bf-8f365481fe98" (UID: "188df47f-f5e6-4222-85bf-8f365481fe98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:34:32.329732 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.329706    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-dshm" (OuterVolumeSpecName: "dshm") pod "188df47f-f5e6-4222-85bf-8f365481fe98" (UID: "188df47f-f5e6-4222-85bf-8f365481fe98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:34:32.329732 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.329719    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188df47f-f5e6-4222-85bf-8f365481fe98-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "188df47f-f5e6-4222-85bf-8f365481fe98" (UID: "188df47f-f5e6-4222-85bf-8f365481fe98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:34:32.329925 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.329895    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188df47f-f5e6-4222-85bf-8f365481fe98-kube-api-access-cxlqv" (OuterVolumeSpecName: "kube-api-access-cxlqv") pod "188df47f-f5e6-4222-85bf-8f365481fe98" (UID: "188df47f-f5e6-4222-85bf-8f365481fe98"). InnerVolumeSpecName "kube-api-access-cxlqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:34:32.330738 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.330716    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4" (OuterVolumeSpecName: "kserve-pvc-source") pod "188df47f-f5e6-4222-85bf-8f365481fe98" (UID: "188df47f-f5e6-4222-85bf-8f365481fe98"). InnerVolumeSpecName "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 15 11:34:32.340534 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.340513    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "188df47f-f5e6-4222-85bf-8f365481fe98" (UID: "188df47f-f5e6-4222-85bf-8f365481fe98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:34:32.427909 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.427878    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:34:32.428073 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.427933    2566 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") on node \"ip-10-0-133-26.ec2.internal\" "
Jul 15 11:34:32.428073 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.427945    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:34:32.428073 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.427954    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxlqv\" (UniqueName: \"kubernetes.io/projected/188df47f-f5e6-4222-85bf-8f365481fe98-kube-api-access-cxlqv\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:34:32.428073 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.427964    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188df47f-f5e6-4222-85bf-8f365481fe98-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:34:32.428073 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.427977    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:34:32.428073 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.427989    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188df47f-f5e6-4222-85bf-8f365481fe98-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:34:32.455073 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.455052    2566 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-441eb1a0-758a-40f3-ad57-542acb4376dc" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4") on node "ip-10-0-133-26.ec2.internal"
Jul 15 11:34:32.529190 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.529156    2566 reconciler_common.go:299] "Volume detached for volume \"pvc-441eb1a0-758a-40f3-ad57-542acb4376dc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03fff1a67ed897ab4\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"csi-28b39141b605ecccebeeed3c7d0a1bb248dc5f88d802eb0dabea42669c8e3558\""
Jul 15 11:34:32.619543 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.619514    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 15 11:34:32.622101 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:32.622073    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 15 11:34:33.268456 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:33.268423    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="188df47f-f5e6-4222-85bf-8f365481fe98" path="/var/lib/kubelet/pods/188df47f-f5e6-4222-85bf-8f365481fe98/volumes"
Jul 15 11:34:38.873900 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:38.873850    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 15 11:34:40.932172 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:40.932136    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"]
Jul 15 11:34:40.932595 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:40.932436    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main"
Jul 15 11:34:40.932595 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:40.932447    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main"
Jul 15 11:34:40.932595 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:40.932468    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="188df47f-f5e6-4222-85bf-8f365481fe98" containerName="main"
Jul 15 11:34:40.932595 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:40.932474    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="188df47f-f5e6-4222-85bf-8f365481fe98" containerName="main"
Jul 15 11:34:40.932595 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:40.932545    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="188df47f-f5e6-4222-85bf-8f365481fe98" containerName="main"
Jul 15 11:34:40.932595 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:40.932553    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eef4ed1-ee90-479f-a819-2e6ea904da05" containerName="main"
Jul 15 11:34:40.934382 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:40.934367    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:40.936967 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:40.936946    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 15 11:34:40.949485 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:40.949465    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"]
Jul 15 11:34:41.006571 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.006534    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-home\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.006752 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.006580    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/755276f7-9ad1-4c60-b725-50fe10932041-tls-certs\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.006752 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.006602    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49rd9\" (UniqueName: \"kubernetes.io/projected/755276f7-9ad1-4c60-b725-50fe10932041-kube-api-access-49rd9\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.006752 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.006680    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-dshm\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.006752 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.006706    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-kserve-provision-location\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.006752 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.006728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-model-cache\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.006946 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.006760    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-tmp-dir\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.107820 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.107777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-dshm\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.107820 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.107831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-kserve-provision-location\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.108079 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.107850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-model-cache\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.108079 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.107980    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-tmp-dir\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.108079 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.108037    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-home\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.108079 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.108064    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/755276f7-9ad1-4c60-b725-50fe10932041-tls-certs\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.108290 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.108091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49rd9\" (UniqueName: \"kubernetes.io/projected/755276f7-9ad1-4c60-b725-50fe10932041-kube-api-access-49rd9\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.108290 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.108231    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-model-cache\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.108385 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.108333    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-tmp-dir\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.108423 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.108385    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-home\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.108469 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.108447    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-kserve-provision-location\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.110400 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.110370    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-dshm\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.110708 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.110687    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/755276f7-9ad1-4c60-b725-50fe10932041-tls-certs\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.117658 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.117633    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49rd9\" (UniqueName: \"kubernetes.io/projected/755276f7-9ad1-4c60-b725-50fe10932041-kube-api-access-49rd9\") pod \"stop-feature-test-kserve-77469bd45-9x49w\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.245974 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.245940    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:41.338225 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.336006    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"]
Jul 15 11:34:41.340096 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.340075    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:34:41.343417 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.343221    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-pw6gx\""
Jul 15 11:34:41.359935 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.359905    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"]
Jul 15 11:34:41.411009 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.410972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4n9n\" (UniqueName: \"kubernetes.io/projected/ddf21f08-52c3-447e-a05b-4a2e26680b07-kube-api-access-h4n9n\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4\" (UID: \"ddf21f08-52c3-447e-a05b-4a2e26680b07\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:34:41.411182 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.411019    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf21f08-52c3-447e-a05b-4a2e26680b07-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4\" (UID: \"ddf21f08-52c3-447e-a05b-4a2e26680b07\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:34:41.411182 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.411143    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ddf21f08-52c3-447e-a05b-4a2e26680b07-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4\" (UID: \"ddf21f08-52c3-447e-a05b-4a2e26680b07\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:34:41.411291 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.411209    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddf21f08-52c3-447e-a05b-4a2e26680b07-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4\" (UID: \"ddf21f08-52c3-447e-a05b-4a2e26680b07\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:34:41.421511 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:34:41.421461    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755276f7_9ad1_4c60_b725_50fe10932041.slice/crio-0c4c0b018815d43ecd22d0416c57fa87b4b44a4a28946872bda4ac60e627a919 WatchSource:0}: Error finding container 0c4c0b018815d43ecd22d0416c57fa87b4b44a4a28946872bda4ac60e627a919: Status 404 returned error can't find the container with id 0c4c0b018815d43ecd22d0416c57fa87b4b44a4a28946872bda4ac60e627a919
Jul 15 11:34:41.429233 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.429206    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"]
Jul 15 11:34:41.512596 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.512483    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddf21f08-52c3-447e-a05b-4a2e26680b07-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4\" (UID: \"ddf21f08-52c3-447e-a05b-4a2e26680b07\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:34:41.512734 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.512636    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4n9n\" (UniqueName: \"kubernetes.io/projected/ddf21f08-52c3-447e-a05b-4a2e26680b07-kube-api-access-h4n9n\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4\" (UID: \"ddf21f08-52c3-447e-a05b-4a2e26680b07\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:34:41.512734 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.512672    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf21f08-52c3-447e-a05b-4a2e26680b07-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4\" (UID: \"ddf21f08-52c3-447e-a05b-4a2e26680b07\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:34:41.512843 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.512822    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddf21f08-52c3-447e-a05b-4a2e26680b07-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4\" (UID: \"ddf21f08-52c3-447e-a05b-4a2e26680b07\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:34:41.512900 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.512882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ddf21f08-52c3-447e-a05b-4a2e26680b07-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4\" (UID: \"ddf21f08-52c3-447e-a05b-4a2e26680b07\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:34:41.513209 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.513189    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ddf21f08-52c3-447e-a05b-4a2e26680b07-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4\" (UID: \"ddf21f08-52c3-447e-a05b-4a2e26680b07\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:34:41.515453 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.515431    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf21f08-52c3-447e-a05b-4a2e26680b07-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4\" (UID: \"ddf21f08-52c3-447e-a05b-4a2e26680b07\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:34:41.525206 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.525185    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4n9n\" (UniqueName: \"kubernetes.io/projected/ddf21f08-52c3-447e-a05b-4a2e26680b07-kube-api-access-h4n9n\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4\" (UID: \"ddf21f08-52c3-447e-a05b-4a2e26680b07\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:34:41.662318 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.662275    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:34:41.806870 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:41.806784    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"]
Jul 15 11:34:41.810651 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:34:41.810620    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf21f08_52c3_447e_a05b_4a2e26680b07.slice/crio-dab3e2da518e9366a43fbb485fdc788a72743690f45b1c1abd7688f8718c7af7 WatchSource:0}: Error finding container dab3e2da518e9366a43fbb485fdc788a72743690f45b1c1abd7688f8718c7af7: Status 404 returned error can't find the container with id dab3e2da518e9366a43fbb485fdc788a72743690f45b1c1abd7688f8718c7af7
Jul 15 11:34:42.345787 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:42.345747    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" event={"ID":"755276f7-9ad1-4c60-b725-50fe10932041","Type":"ContainerStarted","Data":"b88c54334137b732d697b9c161c50661a21082a5b982ea85d50c1c2e49217e8b"}
Jul 15 11:34:42.346377 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:42.345821    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" event={"ID":"755276f7-9ad1-4c60-b725-50fe10932041","Type":"ContainerStarted","Data":"0c4c0b018815d43ecd22d0416c57fa87b4b44a4a28946872bda4ac60e627a919"}
Jul 15 11:34:42.347840 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:42.347815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4" event={"ID":"ddf21f08-52c3-447e-a05b-4a2e26680b07","Type":"ContainerStarted","Data":"a45156621b8f1ab1999acbc682a5e993dcaa7f1cd4fe646207b5cc994f9d7519"}
Jul 15 11:34:42.347955 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:42.347857    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4" event={"ID":"ddf21f08-52c3-447e-a05b-4a2e26680b07","Type":"ContainerStarted","Data":"dab3e2da518e9366a43fbb485fdc788a72743690f45b1c1abd7688f8718c7af7"}
Jul 15 11:34:43.354732 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:43.354695    2566 generic.go:356] "Generic (PLEG): container finished" podID="ddf21f08-52c3-447e-a05b-4a2e26680b07" containerID="a45156621b8f1ab1999acbc682a5e993dcaa7f1cd4fe646207b5cc994f9d7519" exitCode=0
Jul 15 11:34:43.355223 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:43.354779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4" event={"ID":"ddf21f08-52c3-447e-a05b-4a2e26680b07","Type":"ContainerDied","Data":"a45156621b8f1ab1999acbc682a5e993dcaa7f1cd4fe646207b5cc994f9d7519"}
Jul 15 11:34:44.362957 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:44.362925    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4" event={"ID":"ddf21f08-52c3-447e-a05b-4a2e26680b07","Type":"ContainerStarted","Data":"f35534b45cc74ce370e2d6af2229dbdc78ed5fc4aeed3adeb0c3587b7e326f22"}
Jul 15 11:34:44.363379 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:44.363112    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:34:44.384794 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:44.384734    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4" podStartSLOduration=2.633684907 podStartE2EDuration="3.384713787s" totalImagesPullingTime="0s" totalInitContainerRuntime="751.02888ms" isStatefulPod=false podCreationTimestamp="2026-07-15 11:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:34:44.383541188 +0000 UTC m=+2953.613508718" watchObservedRunningTime="2026-07-15 11:34:44.384713787 +0000 UTC m=+2953.614681319"
Jul 15 11:34:45.368748 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:45.368719    2566 generic.go:356] "Generic (PLEG): container finished" podID="755276f7-9ad1-4c60-b725-50fe10932041" containerID="b88c54334137b732d697b9c161c50661a21082a5b982ea85d50c1c2e49217e8b" exitCode=0
Jul 15 11:34:45.369242 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:45.368789    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" event={"ID":"755276f7-9ad1-4c60-b725-50fe10932041","Type":"ContainerDied","Data":"b88c54334137b732d697b9c161c50661a21082a5b982ea85d50c1c2e49217e8b"}
Jul 15 11:34:46.375237 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:46.375204    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" event={"ID":"755276f7-9ad1-4c60-b725-50fe10932041","Type":"ContainerStarted","Data":"21e575043e82a5b35d03c308a95f37105fa73260ba8a7f065f1c42f886fcf816"}
Jul 15 11:34:46.411112 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:46.411048    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" podStartSLOduration=3.544228208 podStartE2EDuration="6.411028696s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.866800488s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:34:46.400016807 +0000 UTC m=+2955.629984339" watchObservedRunningTime="2026-07-15 11:34:46.411028696 +0000 UTC m=+2955.640996226"
Jul 15 11:34:47.246846 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:47.246808    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:48.874083 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:48.874034    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 15 11:34:51.246176 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:51.246143    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:34:51.247817 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:51.247786    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 15 11:34:58.883979 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:58.883937    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:34:58.891655 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:34:58.891629    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:35:01.246889 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:35:01.246834    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 15 11:35:11.246891 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:35:11.246841    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 15 11:35:15.370692 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:35:15.370653    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:35:21.246601 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:35:21.246549    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 15 11:35:31.247090 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:35:31.247042    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 15 11:35:38.431714 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:35:38.431683    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:35:38.432607 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:35:38.432586    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:35:41.247041 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:35:41.246997    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 15 11:35:51.247169 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:35:51.247129    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 15 11:36:01.247354 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:01.247302    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 15 11:36:11.246693 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:11.246647    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 15 11:36:21.246693 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:21.246641    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 15 11:36:31.246360 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:31.246308    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 15 11:36:41.256658 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:41.256627    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:36:41.267433 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:41.267409    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:36:42.791714 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:42.791683    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"]
Jul 15 11:36:43.277105 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:43.277075    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"]
Jul 15 11:36:45.371158 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:45.371122    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4" podUID="ddf21f08-52c3-447e-a05b-4a2e26680b07" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:36:54.160676 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.160644    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"]
Jul 15 11:36:54.163324 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.163308    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:36:54.165992 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.165970    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-9djbw\""
Jul 15 11:36:54.174920 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.174897    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"]
Jul 15 11:36:54.197056 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.197026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7626b4e5-da27-43d1-b87d-323c53209472-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-52fck\" (UID: \"7626b4e5-da27-43d1-b87d-323c53209472\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:36:54.197180 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.197097    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7626b4e5-da27-43d1-b87d-323c53209472-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-52fck\" (UID: \"7626b4e5-da27-43d1-b87d-323c53209472\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:36:54.197180 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.197137    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g265z\" (UniqueName: \"kubernetes.io/projected/7626b4e5-da27-43d1-b87d-323c53209472-kube-api-access-g265z\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-52fck\" (UID: \"7626b4e5-da27-43d1-b87d-323c53209472\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:36:54.197180 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.197157    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7626b4e5-da27-43d1-b87d-323c53209472-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-52fck\" (UID: \"7626b4e5-da27-43d1-b87d-323c53209472\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:36:54.298322 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.298289    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7626b4e5-da27-43d1-b87d-323c53209472-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-52fck\" (UID: \"7626b4e5-da27-43d1-b87d-323c53209472\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:36:54.298488 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.298332    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g265z\" (UniqueName: \"kubernetes.io/projected/7626b4e5-da27-43d1-b87d-323c53209472-kube-api-access-g265z\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-52fck\" (UID: \"7626b4e5-da27-43d1-b87d-323c53209472\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:36:54.298488 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.298352    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7626b4e5-da27-43d1-b87d-323c53209472-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-52fck\" (UID: \"7626b4e5-da27-43d1-b87d-323c53209472\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:36:54.298488 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.298397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7626b4e5-da27-43d1-b87d-323c53209472-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-52fck\" (UID: \"7626b4e5-da27-43d1-b87d-323c53209472\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:36:54.298703 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.298683    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7626b4e5-da27-43d1-b87d-323c53209472-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-52fck\" (UID: \"7626b4e5-da27-43d1-b87d-323c53209472\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:36:54.298750 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.298734    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7626b4e5-da27-43d1-b87d-323c53209472-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-52fck\" (UID: \"7626b4e5-da27-43d1-b87d-323c53209472\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:36:54.300900 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.300876    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7626b4e5-da27-43d1-b87d-323c53209472-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-52fck\" (UID: \"7626b4e5-da27-43d1-b87d-323c53209472\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:36:54.308097 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.308076    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g265z\" (UniqueName: \"kubernetes.io/projected/7626b4e5-da27-43d1-b87d-323c53209472-kube-api-access-g265z\") pod \"stop-feature-test-kserve-router-scheduler-797bcf97c-52fck\" (UID: \"7626b4e5-da27-43d1-b87d-323c53209472\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:36:54.472602 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.472506    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:36:54.608065 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.608042    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"]
Jul 15 11:36:54.610639 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:36:54.610614    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7626b4e5_da27_43d1_b87d_323c53209472.slice/crio-76b1348fd5e2e9a63d0fd9b10817dc48c4b2d728a45183f1f310d3fcc4f43334 WatchSource:0}: Error finding container 76b1348fd5e2e9a63d0fd9b10817dc48c4b2d728a45183f1f310d3fcc4f43334: Status 404 returned error can't find the container with id 76b1348fd5e2e9a63d0fd9b10817dc48c4b2d728a45183f1f310d3fcc4f43334
Jul 15 11:36:54.986011 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.985978    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck" event={"ID":"7626b4e5-da27-43d1-b87d-323c53209472","Type":"ContainerStarted","Data":"db23b2f89028d45682cbfac25938c9a9bd7ed2e4d589b93811e4e782e1618327"}
Jul 15 11:36:54.986184 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:54.986034    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck" event={"ID":"7626b4e5-da27-43d1-b87d-323c53209472","Type":"ContainerStarted","Data":"76b1348fd5e2e9a63d0fd9b10817dc48c4b2d728a45183f1f310d3fcc4f43334"}
Jul 15 11:36:55.992183 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:55.992147    2566 generic.go:356] "Generic (PLEG): container finished" podID="7626b4e5-da27-43d1-b87d-323c53209472" containerID="db23b2f89028d45682cbfac25938c9a9bd7ed2e4d589b93811e4e782e1618327" exitCode=0
Jul 15 11:36:55.992600 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:55.992226    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck" event={"ID":"7626b4e5-da27-43d1-b87d-323c53209472","Type":"ContainerDied","Data":"db23b2f89028d45682cbfac25938c9a9bd7ed2e4d589b93811e4e782e1618327"}
Jul 15 11:36:56.999108 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:56.999070    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck" event={"ID":"7626b4e5-da27-43d1-b87d-323c53209472","Type":"ContainerStarted","Data":"8b094f5830379824de8ef5f01212ede74be11554bf5f922ca94213cec4a32d7a"}
Jul 15 11:36:56.999537 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:56.999176    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:36:57.024285 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:57.024240    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck" podStartSLOduration=2.37564611 podStartE2EDuration="3.024225269s" totalImagesPullingTime="0s" totalInitContainerRuntime="648.579159ms" isStatefulPod=false podCreationTimestamp="2026-07-15 11:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:36:57.022399897 +0000 UTC m=+3086.252367430" watchObservedRunningTime="2026-07-15 11:36:57.024225269 +0000 UTC m=+3086.254192812"
Jul 15 11:36:57.280800 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:57.280724    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 15 11:36:57.951368 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:57.951312    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="main" containerID="cri-o://21e575043e82a5b35d03c308a95f37105fa73260ba8a7f065f1c42f886fcf816" gracePeriod=45
Jul 15 11:36:58.257253 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.257217    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 15 11:36:58.298403 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.298355    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4" podUID="ddf21f08-52c3-447e-a05b-4a2e26680b07" containerName="main" containerID="cri-o://f35534b45cc74ce370e2d6af2229dbdc78ed5fc4aeed3adeb0c3587b7e326f22" gracePeriod=45
Jul 15 11:36:58.573400 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.573375    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:36:58.636039 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.636011    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ddf21f08-52c3-447e-a05b-4a2e26680b07-tokenizer-uds\") pod \"ddf21f08-52c3-447e-a05b-4a2e26680b07\" (UID: \"ddf21f08-52c3-447e-a05b-4a2e26680b07\") "
Jul 15 11:36:58.636185 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.636054    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddf21f08-52c3-447e-a05b-4a2e26680b07-kserve-provision-location\") pod \"ddf21f08-52c3-447e-a05b-4a2e26680b07\" (UID: \"ddf21f08-52c3-447e-a05b-4a2e26680b07\") "
Jul 15 11:36:58.636185 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.636144    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4n9n\" (UniqueName: \"kubernetes.io/projected/ddf21f08-52c3-447e-a05b-4a2e26680b07-kube-api-access-h4n9n\") pod \"ddf21f08-52c3-447e-a05b-4a2e26680b07\" (UID: \"ddf21f08-52c3-447e-a05b-4a2e26680b07\") "
Jul 15 11:36:58.636313 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.636203    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf21f08-52c3-447e-a05b-4a2e26680b07-tls-certs\") pod \"ddf21f08-52c3-447e-a05b-4a2e26680b07\" (UID: \"ddf21f08-52c3-447e-a05b-4a2e26680b07\") "
Jul 15 11:36:58.636367 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.636302    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf21f08-52c3-447e-a05b-4a2e26680b07-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "ddf21f08-52c3-447e-a05b-4a2e26680b07" (UID: "ddf21f08-52c3-447e-a05b-4a2e26680b07"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:36:58.636483 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.636466    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ddf21f08-52c3-447e-a05b-4a2e26680b07-tokenizer-uds\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:36:58.637080 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.636974    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf21f08-52c3-447e-a05b-4a2e26680b07-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ddf21f08-52c3-447e-a05b-4a2e26680b07" (UID: "ddf21f08-52c3-447e-a05b-4a2e26680b07"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:36:58.638861 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.638832    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf21f08-52c3-447e-a05b-4a2e26680b07-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ddf21f08-52c3-447e-a05b-4a2e26680b07" (UID: "ddf21f08-52c3-447e-a05b-4a2e26680b07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:36:58.638979 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.638962    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf21f08-52c3-447e-a05b-4a2e26680b07-kube-api-access-h4n9n" (OuterVolumeSpecName: "kube-api-access-h4n9n") pod "ddf21f08-52c3-447e-a05b-4a2e26680b07" (UID: "ddf21f08-52c3-447e-a05b-4a2e26680b07"). InnerVolumeSpecName "kube-api-access-h4n9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:36:58.694343 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.694320    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:36:58.737050 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.737015    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-dshm\") pod \"755276f7-9ad1-4c60-b725-50fe10932041\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") "
Jul 15 11:36:58.737050 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.737063    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49rd9\" (UniqueName: \"kubernetes.io/projected/755276f7-9ad1-4c60-b725-50fe10932041-kube-api-access-49rd9\") pod \"755276f7-9ad1-4c60-b725-50fe10932041\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") "
Jul 15 11:36:58.737304 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.737092    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/755276f7-9ad1-4c60-b725-50fe10932041-tls-certs\") pod \"755276f7-9ad1-4c60-b725-50fe10932041\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") "
Jul 15 11:36:58.737304 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.737129    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-model-cache\") pod \"755276f7-9ad1-4c60-b725-50fe10932041\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") "
Jul 15 11:36:58.737304 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.737174    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-home\") pod \"755276f7-9ad1-4c60-b725-50fe10932041\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") "
Jul 15 11:36:58.737304 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.737246    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-kserve-provision-location\") pod \"755276f7-9ad1-4c60-b725-50fe10932041\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") "
Jul 15 11:36:58.737304 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.737276    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-tmp-dir\") pod \"755276f7-9ad1-4c60-b725-50fe10932041\" (UID: \"755276f7-9ad1-4c60-b725-50fe10932041\") "
Jul 15 11:36:58.737605 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.737569    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf21f08-52c3-447e-a05b-4a2e26680b07-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:36:58.737605 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.737589    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddf21f08-52c3-447e-a05b-4a2e26680b07-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:36:58.737715 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.737604    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4n9n\" (UniqueName: \"kubernetes.io/projected/ddf21f08-52c3-447e-a05b-4a2e26680b07-kube-api-access-h4n9n\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:36:58.737797 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.737766    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-model-cache" (OuterVolumeSpecName: "model-cache") pod "755276f7-9ad1-4c60-b725-50fe10932041" (UID: "755276f7-9ad1-4c60-b725-50fe10932041"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:36:58.738272 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.738249    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-home" (OuterVolumeSpecName: "home") pod "755276f7-9ad1-4c60-b725-50fe10932041" (UID: "755276f7-9ad1-4c60-b725-50fe10932041"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:36:58.739835 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.739796    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-dshm" (OuterVolumeSpecName: "dshm") pod "755276f7-9ad1-4c60-b725-50fe10932041" (UID: "755276f7-9ad1-4c60-b725-50fe10932041"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:36:58.739835 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.739808    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755276f7-9ad1-4c60-b725-50fe10932041-kube-api-access-49rd9" (OuterVolumeSpecName: "kube-api-access-49rd9") pod "755276f7-9ad1-4c60-b725-50fe10932041" (UID: "755276f7-9ad1-4c60-b725-50fe10932041"). InnerVolumeSpecName "kube-api-access-49rd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:36:58.740461 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.740439    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/755276f7-9ad1-4c60-b725-50fe10932041-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "755276f7-9ad1-4c60-b725-50fe10932041" (UID: "755276f7-9ad1-4c60-b725-50fe10932041"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:36:58.750656 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.750627    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "755276f7-9ad1-4c60-b725-50fe10932041" (UID: "755276f7-9ad1-4c60-b725-50fe10932041"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:36:58.813718 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.813634    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "755276f7-9ad1-4c60-b725-50fe10932041" (UID: "755276f7-9ad1-4c60-b725-50fe10932041"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:36:58.838299 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.838269    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:36:58.838299 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.838295    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49rd9\" (UniqueName: \"kubernetes.io/projected/755276f7-9ad1-4c60-b725-50fe10932041-kube-api-access-49rd9\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:36:58.838463 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.838304    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/755276f7-9ad1-4c60-b725-50fe10932041-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:36:58.838463 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.838313    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:36:58.838463 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.838320    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:36:58.838463 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.838329    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:36:58.838463 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:58.838336    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/755276f7-9ad1-4c60-b725-50fe10932041-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:36:59.011263 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.011229    2566 generic.go:356] "Generic (PLEG): container finished" podID="ddf21f08-52c3-447e-a05b-4a2e26680b07" containerID="f35534b45cc74ce370e2d6af2229dbdc78ed5fc4aeed3adeb0c3587b7e326f22" exitCode=0
Jul 15 11:36:59.011413 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.011280    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"
Jul 15 11:36:59.011413 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.011324    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4" event={"ID":"ddf21f08-52c3-447e-a05b-4a2e26680b07","Type":"ContainerDied","Data":"f35534b45cc74ce370e2d6af2229dbdc78ed5fc4aeed3adeb0c3587b7e326f22"}
Jul 15 11:36:59.011413 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.011376    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4" event={"ID":"ddf21f08-52c3-447e-a05b-4a2e26680b07","Type":"ContainerDied","Data":"dab3e2da518e9366a43fbb485fdc788a72743690f45b1c1abd7688f8718c7af7"}
Jul 15 11:36:59.011413 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.011397    2566 scope.go:117] "RemoveContainer" containerID="f35534b45cc74ce370e2d6af2229dbdc78ed5fc4aeed3adeb0c3587b7e326f22"
Jul 15 11:36:59.013081 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.013045    2566 generic.go:356] "Generic (PLEG): container finished" podID="755276f7-9ad1-4c60-b725-50fe10932041" containerID="21e575043e82a5b35d03c308a95f37105fa73260ba8a7f065f1c42f886fcf816" exitCode=0
Jul 15 11:36:59.013190 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.013085    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"
Jul 15 11:36:59.013190 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.013104    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" event={"ID":"755276f7-9ad1-4c60-b725-50fe10932041","Type":"ContainerDied","Data":"21e575043e82a5b35d03c308a95f37105fa73260ba8a7f065f1c42f886fcf816"}
Jul 15 11:36:59.013190 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.013161    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w" event={"ID":"755276f7-9ad1-4c60-b725-50fe10932041","Type":"ContainerDied","Data":"0c4c0b018815d43ecd22d0416c57fa87b4b44a4a28946872bda4ac60e627a919"}
Jul 15 11:36:59.020248 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.020108    2566 scope.go:117] "RemoveContainer" containerID="a45156621b8f1ab1999acbc682a5e993dcaa7f1cd4fe646207b5cc994f9d7519"
Jul 15 11:36:59.028351 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.028335    2566 scope.go:117] "RemoveContainer" containerID="f35534b45cc74ce370e2d6af2229dbdc78ed5fc4aeed3adeb0c3587b7e326f22"
Jul 15 11:36:59.028629 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:36:59.028611    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35534b45cc74ce370e2d6af2229dbdc78ed5fc4aeed3adeb0c3587b7e326f22\": container with ID starting with f35534b45cc74ce370e2d6af2229dbdc78ed5fc4aeed3adeb0c3587b7e326f22 not found: ID does not exist" containerID="f35534b45cc74ce370e2d6af2229dbdc78ed5fc4aeed3adeb0c3587b7e326f22"
Jul 15 11:36:59.028715 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.028639    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35534b45cc74ce370e2d6af2229dbdc78ed5fc4aeed3adeb0c3587b7e326f22"} err="failed to get container status \"f35534b45cc74ce370e2d6af2229dbdc78ed5fc4aeed3adeb0c3587b7e326f22\": rpc error: code = NotFound desc = could not find container \"f35534b45cc74ce370e2d6af2229dbdc78ed5fc4aeed3adeb0c3587b7e326f22\": container with ID starting with f35534b45cc74ce370e2d6af2229dbdc78ed5fc4aeed3adeb0c3587b7e326f22 not found: ID does not exist"
Jul 15 11:36:59.028715 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.028663    2566 scope.go:117] "RemoveContainer" containerID="a45156621b8f1ab1999acbc682a5e993dcaa7f1cd4fe646207b5cc994f9d7519"
Jul 15 11:36:59.028907 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:36:59.028886    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a45156621b8f1ab1999acbc682a5e993dcaa7f1cd4fe646207b5cc994f9d7519\": container with ID starting with a45156621b8f1ab1999acbc682a5e993dcaa7f1cd4fe646207b5cc994f9d7519 not found: ID does not exist" containerID="a45156621b8f1ab1999acbc682a5e993dcaa7f1cd4fe646207b5cc994f9d7519"
Jul 15 11:36:59.028946 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.028912    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45156621b8f1ab1999acbc682a5e993dcaa7f1cd4fe646207b5cc994f9d7519"} err="failed to get container status \"a45156621b8f1ab1999acbc682a5e993dcaa7f1cd4fe646207b5cc994f9d7519\": rpc error: code = NotFound desc = could not find container \"a45156621b8f1ab1999acbc682a5e993dcaa7f1cd4fe646207b5cc994f9d7519\": container with ID starting with a45156621b8f1ab1999acbc682a5e993dcaa7f1cd4fe646207b5cc994f9d7519 not found: ID does not exist"
Jul 15 11:36:59.028946 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.028927    2566 scope.go:117] "RemoveContainer" containerID="21e575043e82a5b35d03c308a95f37105fa73260ba8a7f065f1c42f886fcf816"
Jul 15 11:36:59.036636 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.036612    2566 scope.go:117] "RemoveContainer" containerID="b88c54334137b732d697b9c161c50661a21082a5b982ea85d50c1c2e49217e8b"
Jul 15 11:36:59.037256 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.037236    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"]
Jul 15 11:36:59.041553 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.041532    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-6jqc4"]
Jul 15 11:36:59.044778 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.044763    2566 scope.go:117] "RemoveContainer" containerID="21e575043e82a5b35d03c308a95f37105fa73260ba8a7f065f1c42f886fcf816"
Jul 15 11:36:59.045011 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:36:59.044993    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e575043e82a5b35d03c308a95f37105fa73260ba8a7f065f1c42f886fcf816\": container with ID starting with 21e575043e82a5b35d03c308a95f37105fa73260ba8a7f065f1c42f886fcf816 not found: ID does not exist" containerID="21e575043e82a5b35d03c308a95f37105fa73260ba8a7f065f1c42f886fcf816"
Jul 15 11:36:59.045094 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.045019    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e575043e82a5b35d03c308a95f37105fa73260ba8a7f065f1c42f886fcf816"} err="failed to get container status \"21e575043e82a5b35d03c308a95f37105fa73260ba8a7f065f1c42f886fcf816\": rpc error: code = NotFound desc = could not find container \"21e575043e82a5b35d03c308a95f37105fa73260ba8a7f065f1c42f886fcf816\": container with ID starting with 21e575043e82a5b35d03c308a95f37105fa73260ba8a7f065f1c42f886fcf816 not found: ID does not exist"
Jul 15 11:36:59.045094 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.045047    2566 scope.go:117] "RemoveContainer" containerID="b88c54334137b732d697b9c161c50661a21082a5b982ea85d50c1c2e49217e8b"
Jul 15 11:36:59.045302 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:36:59.045286    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b88c54334137b732d697b9c161c50661a21082a5b982ea85d50c1c2e49217e8b\": container with ID starting with b88c54334137b732d697b9c161c50661a21082a5b982ea85d50c1c2e49217e8b not found: ID does not exist" containerID="b88c54334137b732d697b9c161c50661a21082a5b982ea85d50c1c2e49217e8b"
Jul 15 11:36:59.045352 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.045308    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b88c54334137b732d697b9c161c50661a21082a5b982ea85d50c1c2e49217e8b"} err="failed to get container status \"b88c54334137b732d697b9c161c50661a21082a5b982ea85d50c1c2e49217e8b\": rpc error: code = NotFound desc = could not find container \"b88c54334137b732d697b9c161c50661a21082a5b982ea85d50c1c2e49217e8b\": container with ID starting with b88c54334137b732d697b9c161c50661a21082a5b982ea85d50c1c2e49217e8b not found: ID does not exist"
Jul 15 11:36:59.054324 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.054304    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"]
Jul 15 11:36:59.058601 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.058575    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-77469bd45-9x49w"]
Jul 15 11:36:59.267325 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.267296    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="755276f7-9ad1-4c60-b725-50fe10932041" path="/var/lib/kubelet/pods/755276f7-9ad1-4c60-b725-50fe10932041/volumes"
Jul 15 11:36:59.267756 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:36:59.267742    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddf21f08-52c3-447e-a05b-4a2e26680b07" path="/var/lib/kubelet/pods/ddf21f08-52c3-447e-a05b-4a2e26680b07/volumes"
Jul 15 11:37:12.304997 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:12.304932    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerName="main" containerID="cri-o://43e55591f5238c44721ebd5d423903f17fee7940980f05504d69c544ae206c71" gracePeriod=45
Jul 15 11:37:12.885259 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:12.885218    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 15 11:37:13.069319 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.069289    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:37:13.077630 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.077561    2566 generic.go:356] "Generic (PLEG): container finished" podID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerID="43e55591f5238c44721ebd5d423903f17fee7940980f05504d69c544ae206c71" exitCode=0
Jul 15 11:37:13.077630 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.077600    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 15 11:37:13.077777 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.077632    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e1508ae0-d9cb-4c7a-83ed-4a73028d5196","Type":"ContainerDied","Data":"43e55591f5238c44721ebd5d423903f17fee7940980f05504d69c544ae206c71"}
Jul 15 11:37:13.077777 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.077691    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e1508ae0-d9cb-4c7a-83ed-4a73028d5196","Type":"ContainerDied","Data":"8dc181b81df0dd059931ea6c54728a2963f043858a36fae0e3ab3e71d3757f73"}
Jul 15 11:37:13.077777 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.077723    2566 scope.go:117] "RemoveContainer" containerID="43e55591f5238c44721ebd5d423903f17fee7940980f05504d69c544ae206c71"
Jul 15 11:37:13.086190 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.086168    2566 scope.go:117] "RemoveContainer" containerID="e34a9fe5c4774842c35563ec9748af925192744af4737afeee7cd35f2195a7b4"
Jul 15 11:37:13.094878 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.094859    2566 scope.go:117] "RemoveContainer" containerID="43e55591f5238c44721ebd5d423903f17fee7940980f05504d69c544ae206c71"
Jul 15 11:37:13.095132 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:37:13.095114    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e55591f5238c44721ebd5d423903f17fee7940980f05504d69c544ae206c71\": container with ID starting with 43e55591f5238c44721ebd5d423903f17fee7940980f05504d69c544ae206c71 not found: ID does not exist" containerID="43e55591f5238c44721ebd5d423903f17fee7940980f05504d69c544ae206c71"
Jul 15 11:37:13.095181 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.095139    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e55591f5238c44721ebd5d423903f17fee7940980f05504d69c544ae206c71"} err="failed to get container status \"43e55591f5238c44721ebd5d423903f17fee7940980f05504d69c544ae206c71\": rpc error: code = NotFound desc = could not find container \"43e55591f5238c44721ebd5d423903f17fee7940980f05504d69c544ae206c71\": container with ID starting with 43e55591f5238c44721ebd5d423903f17fee7940980f05504d69c544ae206c71 not found: ID does not exist"
Jul 15 11:37:13.095181 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.095159    2566 scope.go:117] "RemoveContainer" containerID="e34a9fe5c4774842c35563ec9748af925192744af4737afeee7cd35f2195a7b4"
Jul 15 11:37:13.095381 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:37:13.095352    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34a9fe5c4774842c35563ec9748af925192744af4737afeee7cd35f2195a7b4\": container with ID starting with e34a9fe5c4774842c35563ec9748af925192744af4737afeee7cd35f2195a7b4 not found: ID does not exist" containerID="e34a9fe5c4774842c35563ec9748af925192744af4737afeee7cd35f2195a7b4"
Jul 15 11:37:13.095381 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.095370    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34a9fe5c4774842c35563ec9748af925192744af4737afeee7cd35f2195a7b4"} err="failed to get container status \"e34a9fe5c4774842c35563ec9748af925192744af4737afeee7cd35f2195a7b4\": rpc error: code = NotFound desc = could not find container \"e34a9fe5c4774842c35563ec9748af925192744af4737afeee7cd35f2195a7b4\": container with ID starting with e34a9fe5c4774842c35563ec9748af925192744af4737afeee7cd35f2195a7b4 not found: ID does not exist"
Jul 15 11:37:13.156246 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.156215    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzr7f\" (UniqueName: \"kubernetes.io/projected/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-kube-api-access-zzr7f\") pod \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") "
Jul 15 11:37:13.156390 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.156257    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-kserve-provision-location\") pod \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") "
Jul 15 11:37:13.156390 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.156284    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-dshm\") pod \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") "
Jul 15 11:37:13.156390 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.156324    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-home\") pod \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") "
Jul 15 11:37:13.156390 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.156376    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-tmp-dir\") pod \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") "
Jul 15 11:37:13.156643 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.156400    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-model-cache\") pod \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") "
Jul 15 11:37:13.156643 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.156463    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-tls-certs\") pod \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\" (UID: \"e1508ae0-d9cb-4c7a-83ed-4a73028d5196\") "
Jul 15 11:37:13.156758 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.156713    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1508ae0-d9cb-4c7a-83ed-4a73028d5196" (UID: "e1508ae0-d9cb-4c7a-83ed-4a73028d5196"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:37:13.157018 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.156996    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-home" (OuterVolumeSpecName: "home") pod "e1508ae0-d9cb-4c7a-83ed-4a73028d5196" (UID: "e1508ae0-d9cb-4c7a-83ed-4a73028d5196"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:37:13.158637 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.158608    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-kube-api-access-zzr7f" (OuterVolumeSpecName: "kube-api-access-zzr7f") pod "e1508ae0-d9cb-4c7a-83ed-4a73028d5196" (UID: "e1508ae0-d9cb-4c7a-83ed-4a73028d5196"). InnerVolumeSpecName "kube-api-access-zzr7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:37:13.158996 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.158975    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-dshm" (OuterVolumeSpecName: "dshm") pod "e1508ae0-d9cb-4c7a-83ed-4a73028d5196" (UID: "e1508ae0-d9cb-4c7a-83ed-4a73028d5196"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:37:13.159060 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.159002    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1508ae0-d9cb-4c7a-83ed-4a73028d5196" (UID: "e1508ae0-d9cb-4c7a-83ed-4a73028d5196"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:37:13.173694 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.173661    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1508ae0-d9cb-4c7a-83ed-4a73028d5196" (UID: "e1508ae0-d9cb-4c7a-83ed-4a73028d5196"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:37:13.227468 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.227428    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e1508ae0-d9cb-4c7a-83ed-4a73028d5196" (UID: "e1508ae0-d9cb-4c7a-83ed-4a73028d5196"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:37:13.257368 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.257340    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:37:13.257368 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.257363    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:37:13.257368 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.257372    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:37:13.257609 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.257380    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:37:13.257609 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.257389    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:37:13.257609 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.257397    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzr7f\" (UniqueName: \"kubernetes.io/projected/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-kube-api-access-zzr7f\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:37:13.257609 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.257405    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1508ae0-d9cb-4c7a-83ed-4a73028d5196-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:37:13.394417 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.394343    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 15 11:37:13.400505 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:13.400467    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 15 11:37:15.267299 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:15.267265    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" path="/var/lib/kubelet/pods/e1508ae0-d9cb-4c7a-83ed-4a73028d5196/volumes"
Jul 15 11:37:28.006901 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:37:28.006874    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:38:55.169223 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:38:55.169186    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"]
Jul 15 11:38:58.006606 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:38:58.006571    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck" podUID="7626b4e5-da27-43d1-b87d-323c53209472" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:39:08.007233 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:08.007185    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck" podUID="7626b4e5-da27-43d1-b87d-323c53209472" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:39:10.191926 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.191831    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck" podUID="7626b4e5-da27-43d1-b87d-323c53209472" containerName="main" containerID="cri-o://8b094f5830379824de8ef5f01212ede74be11554bf5f922ca94213cec4a32d7a" gracePeriod=45
Jul 15 11:39:10.452145 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.452083    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:39:10.505582 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.505553    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g265z\" (UniqueName: \"kubernetes.io/projected/7626b4e5-da27-43d1-b87d-323c53209472-kube-api-access-g265z\") pod \"7626b4e5-da27-43d1-b87d-323c53209472\" (UID: \"7626b4e5-da27-43d1-b87d-323c53209472\") "
Jul 15 11:39:10.505738 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.505598    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7626b4e5-da27-43d1-b87d-323c53209472-tls-certs\") pod \"7626b4e5-da27-43d1-b87d-323c53209472\" (UID: \"7626b4e5-da27-43d1-b87d-323c53209472\") "
Jul 15 11:39:10.505738 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.505637    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7626b4e5-da27-43d1-b87d-323c53209472-kserve-provision-location\") pod \"7626b4e5-da27-43d1-b87d-323c53209472\" (UID: \"7626b4e5-da27-43d1-b87d-323c53209472\") "
Jul 15 11:39:10.505825 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.505805    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7626b4e5-da27-43d1-b87d-323c53209472-tokenizer-uds\") pod \"7626b4e5-da27-43d1-b87d-323c53209472\" (UID: \"7626b4e5-da27-43d1-b87d-323c53209472\") "
Jul 15 11:39:10.506127 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.506089    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7626b4e5-da27-43d1-b87d-323c53209472-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "7626b4e5-da27-43d1-b87d-323c53209472" (UID: "7626b4e5-da27-43d1-b87d-323c53209472"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:39:10.506360 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.506340    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7626b4e5-da27-43d1-b87d-323c53209472-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7626b4e5-da27-43d1-b87d-323c53209472" (UID: "7626b4e5-da27-43d1-b87d-323c53209472"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:39:10.507926 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.507908    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7626b4e5-da27-43d1-b87d-323c53209472-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7626b4e5-da27-43d1-b87d-323c53209472" (UID: "7626b4e5-da27-43d1-b87d-323c53209472"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:39:10.507996 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.507928    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7626b4e5-da27-43d1-b87d-323c53209472-kube-api-access-g265z" (OuterVolumeSpecName: "kube-api-access-g265z") pod "7626b4e5-da27-43d1-b87d-323c53209472" (UID: "7626b4e5-da27-43d1-b87d-323c53209472"). InnerVolumeSpecName "kube-api-access-g265z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:39:10.571489 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.571454    2566 generic.go:356] "Generic (PLEG): container finished" podID="7626b4e5-da27-43d1-b87d-323c53209472" containerID="8b094f5830379824de8ef5f01212ede74be11554bf5f922ca94213cec4a32d7a" exitCode=0
Jul 15 11:39:10.571672 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.571527    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"
Jul 15 11:39:10.571672 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.571533    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck" event={"ID":"7626b4e5-da27-43d1-b87d-323c53209472","Type":"ContainerDied","Data":"8b094f5830379824de8ef5f01212ede74be11554bf5f922ca94213cec4a32d7a"}
Jul 15 11:39:10.571672 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.571612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck" event={"ID":"7626b4e5-da27-43d1-b87d-323c53209472","Type":"ContainerDied","Data":"76b1348fd5e2e9a63d0fd9b10817dc48c4b2d728a45183f1f310d3fcc4f43334"}
Jul 15 11:39:10.571672 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.571632    2566 scope.go:117] "RemoveContainer" containerID="8b094f5830379824de8ef5f01212ede74be11554bf5f922ca94213cec4a32d7a"
Jul 15 11:39:10.580757 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.580738    2566 scope.go:117] "RemoveContainer" containerID="db23b2f89028d45682cbfac25938c9a9bd7ed2e4d589b93811e4e782e1618327"
Jul 15 11:39:10.588186 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.588164    2566 scope.go:117] "RemoveContainer" containerID="8b094f5830379824de8ef5f01212ede74be11554bf5f922ca94213cec4a32d7a"
Jul 15 11:39:10.588456 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:39:10.588437    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b094f5830379824de8ef5f01212ede74be11554bf5f922ca94213cec4a32d7a\": container with ID starting with 8b094f5830379824de8ef5f01212ede74be11554bf5f922ca94213cec4a32d7a not found: ID does not exist" containerID="8b094f5830379824de8ef5f01212ede74be11554bf5f922ca94213cec4a32d7a"
Jul 15 11:39:10.588560 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.588464    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b094f5830379824de8ef5f01212ede74be11554bf5f922ca94213cec4a32d7a"} err="failed to get container status \"8b094f5830379824de8ef5f01212ede74be11554bf5f922ca94213cec4a32d7a\": rpc error: code = NotFound desc = could not find container \"8b094f5830379824de8ef5f01212ede74be11554bf5f922ca94213cec4a32d7a\": container with ID starting with 8b094f5830379824de8ef5f01212ede74be11554bf5f922ca94213cec4a32d7a not found: ID does not exist"
Jul 15 11:39:10.588560 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.588481    2566 scope.go:117] "RemoveContainer" containerID="db23b2f89028d45682cbfac25938c9a9bd7ed2e4d589b93811e4e782e1618327"
Jul 15 11:39:10.588889 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:39:10.588869    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db23b2f89028d45682cbfac25938c9a9bd7ed2e4d589b93811e4e782e1618327\": container with ID starting with db23b2f89028d45682cbfac25938c9a9bd7ed2e4d589b93811e4e782e1618327 not found: ID does not exist" containerID="db23b2f89028d45682cbfac25938c9a9bd7ed2e4d589b93811e4e782e1618327"
Jul 15 11:39:10.588941 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.588896    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db23b2f89028d45682cbfac25938c9a9bd7ed2e4d589b93811e4e782e1618327"} err="failed to get container status \"db23b2f89028d45682cbfac25938c9a9bd7ed2e4d589b93811e4e782e1618327\": rpc error: code = NotFound desc = could not find container \"db23b2f89028d45682cbfac25938c9a9bd7ed2e4d589b93811e4e782e1618327\": container with ID starting with db23b2f89028d45682cbfac25938c9a9bd7ed2e4d589b93811e4e782e1618327 not found: ID does not exist"
Jul 15 11:39:10.595410 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.595389    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"]
Jul 15 11:39:10.601842 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.601823    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-797bcf97c-52fck"]
Jul 15 11:39:10.607345 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.607326    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7626b4e5-da27-43d1-b87d-323c53209472-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:39:10.607408 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.607348    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7626b4e5-da27-43d1-b87d-323c53209472-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:39:10.607408 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.607358    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7626b4e5-da27-43d1-b87d-323c53209472-tokenizer-uds\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:39:10.607408 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:10.607367    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g265z\" (UniqueName: \"kubernetes.io/projected/7626b4e5-da27-43d1-b87d-323c53209472-kube-api-access-g265z\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:39:11.267928 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:11.267893    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7626b4e5-da27-43d1-b87d-323c53209472" path="/var/lib/kubelet/pods/7626b4e5-da27-43d1-b87d-323c53209472/volumes"
Jul 15 11:39:29.316118 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316044    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"]
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316330    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="storage-initializer"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316340    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="storage-initializer"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316353    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf21f08-52c3-447e-a05b-4a2e26680b07" containerName="storage-initializer"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316359    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf21f08-52c3-447e-a05b-4a2e26680b07" containerName="storage-initializer"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316366    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="main"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316371    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="main"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316379    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerName="main"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316384    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerName="main"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316392    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7626b4e5-da27-43d1-b87d-323c53209472" containerName="storage-initializer"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316397    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7626b4e5-da27-43d1-b87d-323c53209472" containerName="storage-initializer"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316406    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf21f08-52c3-447e-a05b-4a2e26680b07" containerName="main"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316410    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf21f08-52c3-447e-a05b-4a2e26680b07" containerName="main"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316418    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerName="storage-initializer"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316423    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerName="storage-initializer"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316433    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7626b4e5-da27-43d1-b87d-323c53209472" containerName="main"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316438    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7626b4e5-da27-43d1-b87d-323c53209472" containerName="main"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316518    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1508ae0-d9cb-4c7a-83ed-4a73028d5196" containerName="main"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316527    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddf21f08-52c3-447e-a05b-4a2e26680b07" containerName="main"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316535    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="755276f7-9ad1-4c60-b725-50fe10932041" containerName="main"
Jul 15 11:39:29.316574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.316542    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7626b4e5-da27-43d1-b87d-323c53209472" containerName="main"
Jul 15 11:39:29.318318 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.318301    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.321434 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.321408    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 15 11:39:29.321577 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.321451    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 11:39:29.321577 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.321454    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 11:39:29.321687 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.321650    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 15 11:39:29.335977 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.335956    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"]
Jul 15 11:39:29.349932 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.349907    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.350047 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.349936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8tcc\" (UniqueName: \"kubernetes.io/projected/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-kube-api-access-m8tcc\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.350047 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.349963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-home\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.350168 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.350051    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-model-cache\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.350168 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.350085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-tls-certs\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.350168 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.350149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-dshm\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.350323 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.350173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-cabundle-cert\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.350323 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.350305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-tmp-dir\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.450902 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.450866    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-tmp-dir\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.450902 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.450908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.451133 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.450932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8tcc\" (UniqueName: \"kubernetes.io/projected/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-kube-api-access-m8tcc\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.451133 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.450958    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-home\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.451133 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.450997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-model-cache\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.451133 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.451021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-tls-certs\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.451133 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.451052    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-dshm\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.451133 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.451072    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-cabundle-cert\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.451465 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.451304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.451465 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.451358    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-home\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.451465 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.451430    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-tmp-dir\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.451465 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.451444    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-model-cache\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.451878 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.451813    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-cabundle-cert\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.453625 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.453603    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-dshm\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.453808 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.453793    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-tls-certs\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.459026 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.459005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8tcc\" (UniqueName: \"kubernetes.io/projected/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-kube-api-access-m8tcc\") pod \"lora-single-adapter-test-kserve-8d8476656-tbpf7\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.629529 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.629417    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:29.770127 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:29.770099    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"]
Jul 15 11:39:29.772698 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:39:29.772669    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2d06f5_bd4e_4843_aa8b_d63fa291d8de.slice/crio-c70dccebbf76bd284b22077d7c1e84ba83b7311bbe3f3872de382ffae1c1fd8b WatchSource:0}: Error finding container c70dccebbf76bd284b22077d7c1e84ba83b7311bbe3f3872de382ffae1c1fd8b: Status 404 returned error can't find the container with id c70dccebbf76bd284b22077d7c1e84ba83b7311bbe3f3872de382ffae1c1fd8b
Jul 15 11:39:30.665515 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:30.665471    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7" event={"ID":"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de","Type":"ContainerStarted","Data":"e164dddd1af39a50e6bd4a53e95e9923eef6daac7468f8776f5b5e1bf4e1a99f"}
Jul 15 11:39:30.665916 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:30.665560    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7" event={"ID":"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de","Type":"ContainerStarted","Data":"c70dccebbf76bd284b22077d7c1e84ba83b7311bbe3f3872de382ffae1c1fd8b"}
Jul 15 11:39:33.683949 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:33.683862    2566 generic.go:356] "Generic (PLEG): container finished" podID="3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" containerID="e164dddd1af39a50e6bd4a53e95e9923eef6daac7468f8776f5b5e1bf4e1a99f" exitCode=0
Jul 15 11:39:33.683949 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:33.683939    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7" event={"ID":"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de","Type":"ContainerDied","Data":"e164dddd1af39a50e6bd4a53e95e9923eef6daac7468f8776f5b5e1bf4e1a99f"}
Jul 15 11:39:34.690711 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:34.690670    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7" event={"ID":"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de","Type":"ContainerStarted","Data":"ec0f00fcc8d7a77569dc495c2ffa6a1909acf4e65ec7d2a455bb659005cc0019"}
Jul 15 11:39:34.720130 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:34.720081    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7" podStartSLOduration=2.284175939 podStartE2EDuration="5.720064413s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.435888474s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:39:34.717863783 +0000 UTC m=+3243.947831314" watchObservedRunningTime="2026-07-15 11:39:34.720064413 +0000 UTC m=+3243.950031943"
Jul 15 11:39:35.630547 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:35.630510    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:39.630030 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:39.629996    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:39:39.631713 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:39.631679    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7" podUID="3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 15 11:39:49.629953 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:49.629905    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7" podUID="3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 15 11:39:59.629956 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:39:59.629902    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7" podUID="3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 15 11:40:09.630233 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:40:09.630188    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7" podUID="3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 15 11:40:19.629992 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:40:19.629946    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7" podUID="3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 15 11:40:29.630768 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:40:29.630716    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7" podUID="3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 15 11:40:38.463166 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:40:38.463139    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:40:38.464299 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:40:38.464273    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:40:39.630764 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:40:39.630724    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7" podUID="3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 15 11:40:49.646763 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:40:49.646698    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:40:49.655184 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:40:49.655150    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:41:21.259126 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:21.259092    2566 generic.go:356] "Generic (PLEG): container finished" podID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerID="83521586f29daec561b6d2b25ad55b714de9c32302038bdf0cdfa08b59cca0a3" exitCode=137
Jul 15 11:41:21.259674 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:21.259135    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" event={"ID":"63aad8c8-2687-4adf-9bab-9c36b0086419","Type":"ContainerDied","Data":"83521586f29daec561b6d2b25ad55b714de9c32302038bdf0cdfa08b59cca0a3"}
Jul 15 11:41:21.259674 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:21.259186    2566 scope.go:117] "RemoveContainer" containerID="629e590ed2e52381978bdec15963f9c85350f3b108280dcf4a88baac37f1e057"
Jul 15 11:41:21.259674 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:21.259607    2566 scope.go:117] "RemoveContainer" containerID="83521586f29daec561b6d2b25ad55b714de9c32302038bdf0cdfa08b59cca0a3"
Jul 15 11:41:21.260707 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:21.260674    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:41:21.260812 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:21.260760    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:41:21.260855 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:21.260807    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 15 11:41:22.265758 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:22.265721    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" event={"ID":"63aad8c8-2687-4adf-9bab-9c36b0086419","Type":"ContainerStarted","Data":"a0d1a7c0d52ec9ff1fafce782d84dcd2597e86470f208cf7d6b95bc3e955e744"}
Jul 15 11:41:22.797820 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:22.797782    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:41:22.799117 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:22.799091    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:41:22.799251 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:22.799157    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:41:23.424600 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:23.424568    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:41:23.798330 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:23.798289    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:41:23.798559 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:23.798354    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:41:24.636717 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:24.636684    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"]
Jul 15 11:41:24.640089 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:24.640072    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"
Jul 15 11:41:24.644364 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:24.644345    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-2hnl6\""
Jul 15 11:41:24.644517 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:24.644381    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 15 11:41:24.656674 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:24.656652    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"]
Jul 15 11:41:24.720049 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:24.719991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9lv8\" (UniqueName: \"kubernetes.io/projected/9aeab130-f915-4a29-9dee-34aed77160b3-kube-api-access-l9lv8\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n\" (UID: \"9aeab130-f915-4a29-9dee-34aed77160b3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"
Jul 15 11:41:24.720431 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:24.720078    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aeab130-f915-4a29-9dee-34aed77160b3-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n\" (UID: \"9aeab130-f915-4a29-9dee-34aed77160b3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"
Jul 15 11:41:24.720431 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:24.720123    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9aeab130-f915-4a29-9dee-34aed77160b3-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n\" (UID: \"9aeab130-f915-4a29-9dee-34aed77160b3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"
Jul 15 11:41:24.798415 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:24.798378    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:41:24.798622 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:24.798434    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:41:24.821010 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:24.820970    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9lv8\" (UniqueName: \"kubernetes.io/projected/9aeab130-f915-4a29-9dee-34aed77160b3-kube-api-access-l9lv8\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n\" (UID: \"9aeab130-f915-4a29-9dee-34aed77160b3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"
Jul 15 11:41:24.821168 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:24.821054    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aeab130-f915-4a29-9dee-34aed77160b3-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n\" (UID: \"9aeab130-f915-4a29-9dee-34aed77160b3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"
Jul 15 11:41:24.821168 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:24.821102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9aeab130-f915-4a29-9dee-34aed77160b3-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n\" (UID: \"9aeab130-f915-4a29-9dee-34aed77160b3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"
Jul 15 11:41:24.821463 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:24.821441    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9aeab130-f915-4a29-9dee-34aed77160b3-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n\" (UID: \"9aeab130-f915-4a29-9dee-34aed77160b3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"
Jul 15 11:41:24.823856 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:24.823831    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aeab130-f915-4a29-9dee-34aed77160b3-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n\" (UID: \"9aeab130-f915-4a29-9dee-34aed77160b3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"
Jul 15 11:41:24.830180 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:24.830156    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9lv8\" (UniqueName: \"kubernetes.io/projected/9aeab130-f915-4a29-9dee-34aed77160b3-kube-api-access-l9lv8\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n\" (UID: \"9aeab130-f915-4a29-9dee-34aed77160b3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"
Jul 15 11:41:24.951553 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:24.951440    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"
Jul 15 11:41:25.134455 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:25.134301    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"]
Jul 15 11:41:25.137025 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:41:25.136994    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aeab130_f915_4a29_9dee_34aed77160b3.slice/crio-c4997cbf6dfca1271c623e845820326834d540889d90d23d051c96e84eba9733 WatchSource:0}: Error finding container c4997cbf6dfca1271c623e845820326834d540889d90d23d051c96e84eba9733: Status 404 returned error can't find the container with id c4997cbf6dfca1271c623e845820326834d540889d90d23d051c96e84eba9733
Jul 15 11:41:25.285801 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:25.285755    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n" event={"ID":"9aeab130-f915-4a29-9dee-34aed77160b3","Type":"ContainerStarted","Data":"23f9afbe951397fc214e063abcadfc225615bb7c7845df757990d24b6e3f71b8"}
Jul 15 11:41:25.286002 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:25.285826    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n" event={"ID":"9aeab130-f915-4a29-9dee-34aed77160b3","Type":"ContainerStarted","Data":"c4997cbf6dfca1271c623e845820326834d540889d90d23d051c96e84eba9733"}
Jul 15 11:41:25.286633 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:25.286608    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"
Jul 15 11:41:25.311847 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:25.311796    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n" podStartSLOduration=1.311781566 podStartE2EDuration="1.311781566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:41:25.309658524 +0000 UTC m=+3354.539626058" watchObservedRunningTime="2026-07-15 11:41:25.311781566 +0000 UTC m=+3354.541749097"
Jul 15 11:41:25.798446 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:25.798404    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:41:25.798894 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:25.798465    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:41:26.798068 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:26.798031    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-fmhhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Jul 15 11:41:26.798250 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:26.798088    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm" podUID="63aad8c8-2687-4adf-9bab-9c36b0086419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Jul 15 11:41:28.730456 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:28.730412    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:41:28.731821 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:28.731800    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-fmhhm"
Jul 15 11:41:57.299345 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:57.299317    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"
Jul 15 11:41:59.114240 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:41:59.114210    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"]
Jul 15 11:42:07.299454 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:07.299417    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n" podUID="9aeab130-f915-4a29-9dee-34aed77160b3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:42:14.141553 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.141479    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n" podUID="9aeab130-f915-4a29-9dee-34aed77160b3" containerName="main" containerID="cri-o://23f9afbe951397fc214e063abcadfc225615bb7c7845df757990d24b6e3f71b8" gracePeriod=45
Jul 15 11:42:14.402720 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.402665    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"
Jul 15 11:42:14.510807 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.510775    2566 generic.go:356] "Generic (PLEG): container finished" podID="9aeab130-f915-4a29-9dee-34aed77160b3" containerID="23f9afbe951397fc214e063abcadfc225615bb7c7845df757990d24b6e3f71b8" exitCode=0
Jul 15 11:42:14.511001 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.510822    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"
Jul 15 11:42:14.511001 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.510829    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n" event={"ID":"9aeab130-f915-4a29-9dee-34aed77160b3","Type":"ContainerDied","Data":"23f9afbe951397fc214e063abcadfc225615bb7c7845df757990d24b6e3f71b8"}
Jul 15 11:42:14.511001 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.510881    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n" event={"ID":"9aeab130-f915-4a29-9dee-34aed77160b3","Type":"ContainerDied","Data":"c4997cbf6dfca1271c623e845820326834d540889d90d23d051c96e84eba9733"}
Jul 15 11:42:14.511001 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.510906    2566 scope.go:117] "RemoveContainer" containerID="23f9afbe951397fc214e063abcadfc225615bb7c7845df757990d24b6e3f71b8"
Jul 15 11:42:14.519137 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.519121    2566 scope.go:117] "RemoveContainer" containerID="23f9afbe951397fc214e063abcadfc225615bb7c7845df757990d24b6e3f71b8"
Jul 15 11:42:14.519416 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:42:14.519399    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23f9afbe951397fc214e063abcadfc225615bb7c7845df757990d24b6e3f71b8\": container with ID starting with 23f9afbe951397fc214e063abcadfc225615bb7c7845df757990d24b6e3f71b8 not found: ID does not exist" containerID="23f9afbe951397fc214e063abcadfc225615bb7c7845df757990d24b6e3f71b8"
Jul 15 11:42:14.519456 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.519425    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f9afbe951397fc214e063abcadfc225615bb7c7845df757990d24b6e3f71b8"} err="failed to get container status \"23f9afbe951397fc214e063abcadfc225615bb7c7845df757990d24b6e3f71b8\": rpc error: code = NotFound desc = could not find container \"23f9afbe951397fc214e063abcadfc225615bb7c7845df757990d24b6e3f71b8\": container with ID starting with 23f9afbe951397fc214e063abcadfc225615bb7c7845df757990d24b6e3f71b8 not found: ID does not exist"
Jul 15 11:42:14.566908 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.566881    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9lv8\" (UniqueName: \"kubernetes.io/projected/9aeab130-f915-4a29-9dee-34aed77160b3-kube-api-access-l9lv8\") pod \"9aeab130-f915-4a29-9dee-34aed77160b3\" (UID: \"9aeab130-f915-4a29-9dee-34aed77160b3\") "
Jul 15 11:42:14.567107 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.566942    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9aeab130-f915-4a29-9dee-34aed77160b3-tokenizer-uds\") pod \"9aeab130-f915-4a29-9dee-34aed77160b3\" (UID: \"9aeab130-f915-4a29-9dee-34aed77160b3\") "
Jul 15 11:42:14.567107 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.566981    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aeab130-f915-4a29-9dee-34aed77160b3-tls-certs\") pod \"9aeab130-f915-4a29-9dee-34aed77160b3\" (UID: \"9aeab130-f915-4a29-9dee-34aed77160b3\") "
Jul 15 11:42:14.567290 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.567268    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aeab130-f915-4a29-9dee-34aed77160b3-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "9aeab130-f915-4a29-9dee-34aed77160b3" (UID: "9aeab130-f915-4a29-9dee-34aed77160b3"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:42:14.569204 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.569172    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aeab130-f915-4a29-9dee-34aed77160b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9aeab130-f915-4a29-9dee-34aed77160b3" (UID: "9aeab130-f915-4a29-9dee-34aed77160b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:42:14.569204 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.569196    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aeab130-f915-4a29-9dee-34aed77160b3-kube-api-access-l9lv8" (OuterVolumeSpecName: "kube-api-access-l9lv8") pod "9aeab130-f915-4a29-9dee-34aed77160b3" (UID: "9aeab130-f915-4a29-9dee-34aed77160b3"). InnerVolumeSpecName "kube-api-access-l9lv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:42:14.668274 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.668165    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9aeab130-f915-4a29-9dee-34aed77160b3-tokenizer-uds\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:42:14.668274 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.668205    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aeab130-f915-4a29-9dee-34aed77160b3-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:42:14.668274 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.668218    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9lv8\" (UniqueName: \"kubernetes.io/projected/9aeab130-f915-4a29-9dee-34aed77160b3-kube-api-access-l9lv8\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:42:14.834290 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.834254    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"]
Jul 15 11:42:14.839576 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:14.839531    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffpxd9n"]
Jul 15 11:42:15.267662 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:15.267631    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aeab130-f915-4a29-9dee-34aed77160b3" path="/var/lib/kubelet/pods/9aeab130-f915-4a29-9dee-34aed77160b3/volumes"
Jul 15 11:42:31.517149 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.517059    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"]
Jul 15 11:42:31.517768 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.517534    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aeab130-f915-4a29-9dee-34aed77160b3" containerName="main"
Jul 15 11:42:31.517768 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.517552    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aeab130-f915-4a29-9dee-34aed77160b3" containerName="main"
Jul 15 11:42:31.517768 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.517660    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aeab130-f915-4a29-9dee-34aed77160b3" containerName="main"
Jul 15 11:42:31.522551 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.522525    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"
Jul 15 11:42:31.525927 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.525884    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-6qmnm\""
Jul 15 11:42:31.525927 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.525898    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 15 11:42:31.534047 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.534022    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"]
Jul 15 11:42:31.605812 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.605782    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbmch\" (UniqueName: \"kubernetes.io/projected/50d9bf0d-c401-470c-a603-c6b8988cfad6-kube-api-access-cbmch\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b\" (UID: \"50d9bf0d-c401-470c-a603-c6b8988cfad6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"
Jul 15 11:42:31.605972 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.605844    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/50d9bf0d-c401-470c-a603-c6b8988cfad6-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b\" (UID: \"50d9bf0d-c401-470c-a603-c6b8988cfad6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"
Jul 15 11:42:31.605972 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.605874    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50d9bf0d-c401-470c-a603-c6b8988cfad6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b\" (UID: \"50d9bf0d-c401-470c-a603-c6b8988cfad6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"
Jul 15 11:42:31.706878 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.706839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbmch\" (UniqueName: \"kubernetes.io/projected/50d9bf0d-c401-470c-a603-c6b8988cfad6-kube-api-access-cbmch\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b\" (UID: \"50d9bf0d-c401-470c-a603-c6b8988cfad6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"
Jul 15 11:42:31.707059 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.706904    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/50d9bf0d-c401-470c-a603-c6b8988cfad6-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b\" (UID: \"50d9bf0d-c401-470c-a603-c6b8988cfad6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"
Jul 15 11:42:31.707059 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.706931    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50d9bf0d-c401-470c-a603-c6b8988cfad6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b\" (UID: \"50d9bf0d-c401-470c-a603-c6b8988cfad6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"
Jul 15 11:42:31.707317 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.707296    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/50d9bf0d-c401-470c-a603-c6b8988cfad6-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b\" (UID: \"50d9bf0d-c401-470c-a603-c6b8988cfad6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"
Jul 15 11:42:31.709527 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.709478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50d9bf0d-c401-470c-a603-c6b8988cfad6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b\" (UID: \"50d9bf0d-c401-470c-a603-c6b8988cfad6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"
Jul 15 11:42:31.714954 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.714927    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbmch\" (UniqueName: \"kubernetes.io/projected/50d9bf0d-c401-470c-a603-c6b8988cfad6-kube-api-access-cbmch\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b\" (UID: \"50d9bf0d-c401-470c-a603-c6b8988cfad6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"
Jul 15 11:42:31.836857 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.836773    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"
Jul 15 11:42:31.969085 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:31.968862    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"]
Jul 15 11:42:31.971842 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:42:31.971814    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d9bf0d_c401_470c_a603_c6b8988cfad6.slice/crio-b843567d0a83fa34cdb9564608c916f44a6f56b96b8a0a3b2d429da63d684d95 WatchSource:0}: Error finding container b843567d0a83fa34cdb9564608c916f44a6f56b96b8a0a3b2d429da63d684d95: Status 404 returned error can't find the container with id b843567d0a83fa34cdb9564608c916f44a6f56b96b8a0a3b2d429da63d684d95
Jul 15 11:42:32.599125 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:32.599090    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b" event={"ID":"50d9bf0d-c401-470c-a603-c6b8988cfad6","Type":"ContainerStarted","Data":"bbcc432a0650d4fd9f17ff9a572a8a08d55b8e8e89d4f0805d5af012e5f31bde"}
Jul 15 11:42:32.599539 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:32.599143    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b" event={"ID":"50d9bf0d-c401-470c-a603-c6b8988cfad6","Type":"ContainerStarted","Data":"b843567d0a83fa34cdb9564608c916f44a6f56b96b8a0a3b2d429da63d684d95"}
Jul 15 11:42:32.599539 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:32.599169    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"
Jul 15 11:42:32.618260 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:42:32.618217    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b" podStartSLOduration=1.6182035460000002 podStartE2EDuration="1.618203546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:42:32.616602676 +0000 UTC m=+3421.846570219" watchObservedRunningTime="2026-07-15 11:42:32.618203546 +0000 UTC m=+3421.848171077"
Jul 15 11:43:03.604629 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:03.604601    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"
Jul 15 11:43:42.928826 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:42.928782    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"]
Jul 15 11:43:57.952274 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:57.952161    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7" podUID="3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" containerName="main" containerID="cri-o://ec0f00fcc8d7a77569dc495c2ffa6a1909acf4e65ec7d2a455bb659005cc0019" gracePeriod=45
Jul 15 11:43:58.616813 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.616783    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:43:58.786648 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.786615    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-cabundle-cert\") pod \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") "
Jul 15 11:43:58.786828 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.786663    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-tls-certs\") pod \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") "
Jul 15 11:43:58.786828 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.786690    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-kserve-provision-location\") pod \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") "
Jul 15 11:43:58.786828 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.786718    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-home\") pod \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") "
Jul 15 11:43:58.786828 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.786742    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8tcc\" (UniqueName: \"kubernetes.io/projected/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-kube-api-access-m8tcc\") pod \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") "
Jul 15 11:43:58.786828 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.786787    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-model-cache\") pod \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") "
Jul 15 11:43:58.786828 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.786814    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-dshm\") pod \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") "
Jul 15 11:43:58.787159 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.786845    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-tmp-dir\") pod \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\" (UID: \"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de\") "
Jul 15 11:43:58.787159 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.787046    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" (UID: "3d2d06f5-bd4e-4843-aa8b-d63fa291d8de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:43:58.787250 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.787170    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-home" (OuterVolumeSpecName: "home") pod "3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" (UID: "3d2d06f5-bd4e-4843-aa8b-d63fa291d8de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:43:58.787364 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.787341    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" (UID: "3d2d06f5-bd4e-4843-aa8b-d63fa291d8de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 15 11:43:58.789273 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.789240    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-dshm" (OuterVolumeSpecName: "dshm") pod "3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" (UID: "3d2d06f5-bd4e-4843-aa8b-d63fa291d8de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:43:58.789605 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.789574    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-kube-api-access-m8tcc" (OuterVolumeSpecName: "kube-api-access-m8tcc") pod "3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" (UID: "3d2d06f5-bd4e-4843-aa8b-d63fa291d8de"). InnerVolumeSpecName "kube-api-access-m8tcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:43:58.789696 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.789662    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" (UID: "3d2d06f5-bd4e-4843-aa8b-d63fa291d8de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:43:58.799187 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.799164    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" (UID: "3d2d06f5-bd4e-4843-aa8b-d63fa291d8de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:43:58.856745 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.856711    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" (UID: "3d2d06f5-bd4e-4843-aa8b-d63fa291d8de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:43:58.888228 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.888195    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:43:58.888228 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.888226    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:43:58.888422 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.888238    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:43:58.888422 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.888250    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8tcc\" (UniqueName: \"kubernetes.io/projected/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-kube-api-access-m8tcc\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:43:58.888422 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.888261    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:43:58.888422 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.888271    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:43:58.888422 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.888280    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:43:58.888422 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:58.888290    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de-cabundle-cert\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:43:59.003866 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:59.003837    2566 generic.go:356] "Generic (PLEG): container finished" podID="3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" containerID="ec0f00fcc8d7a77569dc495c2ffa6a1909acf4e65ec7d2a455bb659005cc0019" exitCode=0
Jul 15 11:43:59.004247 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:59.003892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7" event={"ID":"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de","Type":"ContainerDied","Data":"ec0f00fcc8d7a77569dc495c2ffa6a1909acf4e65ec7d2a455bb659005cc0019"}
Jul 15 11:43:59.004247 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:59.003899    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"
Jul 15 11:43:59.004247 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:59.003935    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7" event={"ID":"3d2d06f5-bd4e-4843-aa8b-d63fa291d8de","Type":"ContainerDied","Data":"c70dccebbf76bd284b22077d7c1e84ba83b7311bbe3f3872de382ffae1c1fd8b"}
Jul 15 11:43:59.004247 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:59.003958    2566 scope.go:117] "RemoveContainer" containerID="ec0f00fcc8d7a77569dc495c2ffa6a1909acf4e65ec7d2a455bb659005cc0019"
Jul 15 11:43:59.013332 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:59.013316    2566 scope.go:117] "RemoveContainer" containerID="e164dddd1af39a50e6bd4a53e95e9923eef6daac7468f8776f5b5e1bf4e1a99f"
Jul 15 11:43:59.020975 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:59.020958    2566 scope.go:117] "RemoveContainer" containerID="ec0f00fcc8d7a77569dc495c2ffa6a1909acf4e65ec7d2a455bb659005cc0019"
Jul 15 11:43:59.021232 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:43:59.021214    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec0f00fcc8d7a77569dc495c2ffa6a1909acf4e65ec7d2a455bb659005cc0019\": container with ID starting with ec0f00fcc8d7a77569dc495c2ffa6a1909acf4e65ec7d2a455bb659005cc0019 not found: ID does not exist" containerID="ec0f00fcc8d7a77569dc495c2ffa6a1909acf4e65ec7d2a455bb659005cc0019"
Jul 15 11:43:59.021305 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:59.021244    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec0f00fcc8d7a77569dc495c2ffa6a1909acf4e65ec7d2a455bb659005cc0019"} err="failed to get container status \"ec0f00fcc8d7a77569dc495c2ffa6a1909acf4e65ec7d2a455bb659005cc0019\": rpc error: code = NotFound desc = could not find container \"ec0f00fcc8d7a77569dc495c2ffa6a1909acf4e65ec7d2a455bb659005cc0019\": container with ID starting with ec0f00fcc8d7a77569dc495c2ffa6a1909acf4e65ec7d2a455bb659005cc0019 not found: ID does not exist"
Jul 15 11:43:59.021305 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:59.021268    2566 scope.go:117] "RemoveContainer" containerID="e164dddd1af39a50e6bd4a53e95e9923eef6daac7468f8776f5b5e1bf4e1a99f"
Jul 15 11:43:59.021515 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:43:59.021482    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e164dddd1af39a50e6bd4a53e95e9923eef6daac7468f8776f5b5e1bf4e1a99f\": container with ID starting with e164dddd1af39a50e6bd4a53e95e9923eef6daac7468f8776f5b5e1bf4e1a99f not found: ID does not exist" containerID="e164dddd1af39a50e6bd4a53e95e9923eef6daac7468f8776f5b5e1bf4e1a99f"
Jul 15 11:43:59.021567 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:59.021520    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e164dddd1af39a50e6bd4a53e95e9923eef6daac7468f8776f5b5e1bf4e1a99f"} err="failed to get container status \"e164dddd1af39a50e6bd4a53e95e9923eef6daac7468f8776f5b5e1bf4e1a99f\": rpc error: code = NotFound desc = could not find container \"e164dddd1af39a50e6bd4a53e95e9923eef6daac7468f8776f5b5e1bf4e1a99f\": container with ID starting with e164dddd1af39a50e6bd4a53e95e9923eef6daac7468f8776f5b5e1bf4e1a99f not found: ID does not exist"
Jul 15 11:43:59.027843 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:59.027820    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"]
Jul 15 11:43:59.031458 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:59.031437    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-8d8476656-tbpf7"]
Jul 15 11:43:59.267127 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:43:59.267101    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" path="/var/lib/kubelet/pods/3d2d06f5-bd4e-4843-aa8b-d63fa291d8de/volumes"
Jul 15 11:44:25.761649 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:25.761605    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"]
Jul 15 11:44:33.604785 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:33.604751    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b" podUID="50d9bf0d-c401-470c-a603-c6b8988cfad6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:44:36.580942 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:36.580902    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b" message="" err="command '/bin/sleep 15' exited with 137: "
Jul 15 11:44:36.581203 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:44:36.580945    2566 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b" podUID="50d9bf0d-c401-470c-a603-c6b8988cfad6" containerName="main" containerID="cri-o://bbcc432a0650d4fd9f17ff9a572a8a08d55b8e8e89d4f0805d5af012e5f31bde"
Jul 15 11:44:36.581203 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:36.580988    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b" podUID="50d9bf0d-c401-470c-a603-c6b8988cfad6" containerName="main" containerID="cri-o://bbcc432a0650d4fd9f17ff9a572a8a08d55b8e8e89d4f0805d5af012e5f31bde" gracePeriod=50
Jul 15 11:44:36.733448 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:36.733423    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b_50d9bf0d-c401-470c-a603-c6b8988cfad6/main/0.log"
Jul 15 11:44:36.735395 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:36.735373    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"
Jul 15 11:44:36.810760 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:36.810731    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50d9bf0d-c401-470c-a603-c6b8988cfad6-tls-certs\") pod \"50d9bf0d-c401-470c-a603-c6b8988cfad6\" (UID: \"50d9bf0d-c401-470c-a603-c6b8988cfad6\") "
Jul 15 11:44:36.810907 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:36.810770    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/50d9bf0d-c401-470c-a603-c6b8988cfad6-tokenizer-uds\") pod \"50d9bf0d-c401-470c-a603-c6b8988cfad6\" (UID: \"50d9bf0d-c401-470c-a603-c6b8988cfad6\") "
Jul 15 11:44:36.810907 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:36.810817    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbmch\" (UniqueName: \"kubernetes.io/projected/50d9bf0d-c401-470c-a603-c6b8988cfad6-kube-api-access-cbmch\") pod \"50d9bf0d-c401-470c-a603-c6b8988cfad6\" (UID: \"50d9bf0d-c401-470c-a603-c6b8988cfad6\") "
Jul 15 11:44:36.811070 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:36.811046    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d9bf0d-c401-470c-a603-c6b8988cfad6-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "50d9bf0d-c401-470c-a603-c6b8988cfad6" (UID: "50d9bf0d-c401-470c-a603-c6b8988cfad6"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:44:36.812958 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:36.812937    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50d9bf0d-c401-470c-a603-c6b8988cfad6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50d9bf0d-c401-470c-a603-c6b8988cfad6" (UID: "50d9bf0d-c401-470c-a603-c6b8988cfad6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:44:36.813048 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:36.812971    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d9bf0d-c401-470c-a603-c6b8988cfad6-kube-api-access-cbmch" (OuterVolumeSpecName: "kube-api-access-cbmch") pod "50d9bf0d-c401-470c-a603-c6b8988cfad6" (UID: "50d9bf0d-c401-470c-a603-c6b8988cfad6"). InnerVolumeSpecName "kube-api-access-cbmch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:44:36.912401 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:36.912313    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbmch\" (UniqueName: \"kubernetes.io/projected/50d9bf0d-c401-470c-a603-c6b8988cfad6-kube-api-access-cbmch\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:44:36.912401 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:36.912345    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50d9bf0d-c401-470c-a603-c6b8988cfad6-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:44:36.912401 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:36.912359    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/50d9bf0d-c401-470c-a603-c6b8988cfad6-tokenizer-uds\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:44:37.175220 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:37.175137    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b_50d9bf0d-c401-470c-a603-c6b8988cfad6/main/0.log"
Jul 15 11:44:37.175220 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:37.175183    2566 generic.go:356] "Generic (PLEG): container finished" podID="50d9bf0d-c401-470c-a603-c6b8988cfad6" containerID="bbcc432a0650d4fd9f17ff9a572a8a08d55b8e8e89d4f0805d5af012e5f31bde" exitCode=1
Jul 15 11:44:37.175484 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:37.175218    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b" event={"ID":"50d9bf0d-c401-470c-a603-c6b8988cfad6","Type":"ContainerDied","Data":"bbcc432a0650d4fd9f17ff9a572a8a08d55b8e8e89d4f0805d5af012e5f31bde"}
Jul 15 11:44:37.175484 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:37.175243    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"
Jul 15 11:44:37.175484 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:37.175270    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b" event={"ID":"50d9bf0d-c401-470c-a603-c6b8988cfad6","Type":"ContainerDied","Data":"b843567d0a83fa34cdb9564608c916f44a6f56b96b8a0a3b2d429da63d684d95"}
Jul 15 11:44:37.175484 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:37.175294    2566 scope.go:117] "RemoveContainer" containerID="bbcc432a0650d4fd9f17ff9a572a8a08d55b8e8e89d4f0805d5af012e5f31bde"
Jul 15 11:44:37.184946 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:37.184929    2566 scope.go:117] "RemoveContainer" containerID="bbcc432a0650d4fd9f17ff9a572a8a08d55b8e8e89d4f0805d5af012e5f31bde"
Jul 15 11:44:37.185259 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:44:37.185237    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbcc432a0650d4fd9f17ff9a572a8a08d55b8e8e89d4f0805d5af012e5f31bde\": container with ID starting with bbcc432a0650d4fd9f17ff9a572a8a08d55b8e8e89d4f0805d5af012e5f31bde not found: ID does not exist" containerID="bbcc432a0650d4fd9f17ff9a572a8a08d55b8e8e89d4f0805d5af012e5f31bde"
Jul 15 11:44:37.185330 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:37.185269    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbcc432a0650d4fd9f17ff9a572a8a08d55b8e8e89d4f0805d5af012e5f31bde"} err="failed to get container status \"bbcc432a0650d4fd9f17ff9a572a8a08d55b8e8e89d4f0805d5af012e5f31bde\": rpc error: code = NotFound desc = could not find container \"bbcc432a0650d4fd9f17ff9a572a8a08d55b8e8e89d4f0805d5af012e5f31bde\": container with ID starting with bbcc432a0650d4fd9f17ff9a572a8a08d55b8e8e89d4f0805d5af012e5f31bde not found: ID does not exist"
Jul 15 11:44:37.203162 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:37.203133    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"]
Jul 15 11:44:37.208717 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:37.208690    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fpxd4b"]
Jul 15 11:44:37.273517 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:37.273461    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50d9bf0d-c401-470c-a603-c6b8988cfad6" path="/var/lib/kubelet/pods/50d9bf0d-c401-470c-a603-c6b8988cfad6/volumes"
Jul 15 11:44:59.996874 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:59.996835    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"]
Jul 15 11:44:59.997529 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:59.997150    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" containerName="main"
Jul 15 11:44:59.997529 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:59.997162    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" containerName="main"
Jul 15 11:44:59.997529 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:59.997177    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d9bf0d-c401-470c-a603-c6b8988cfad6" containerName="main"
Jul 15 11:44:59.997529 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:59.997183    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d9bf0d-c401-470c-a603-c6b8988cfad6" containerName="main"
Jul 15 11:44:59.997529 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:59.997190    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" containerName="storage-initializer"
Jul 15 11:44:59.997529 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:59.997195    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" containerName="storage-initializer"
Jul 15 11:44:59.997529 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:59.997248    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d9bf0d-c401-470c-a603-c6b8988cfad6" containerName="main"
Jul 15 11:44:59.997529 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:44:59.997257    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d2d06f5-bd4e-4843-aa8b-d63fa291d8de" containerName="main"
Jul 15 11:45:00.000177 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.000155    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.003868 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.003830    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 15 11:45:00.004020 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.003913    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 11:45:00.018679 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.014812    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"]
Jul 15 11:45:00.025020 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.019653    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.025020 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.019731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.025020 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.019767    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.025020 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.019860    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.025020 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.019907    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.025020 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.019945    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcrdm\" (UniqueName: \"kubernetes.io/projected/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-kube-api-access-rcrdm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.120866 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.120825    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.120866 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.120869    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.121121 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.120895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcrdm\" (UniqueName: \"kubernetes.io/projected/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-kube-api-access-rcrdm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.121121 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.120934    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.121121 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.120969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.121121 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.120995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.121425 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.121393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.121425 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.121409    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.121628 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.121434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.123215 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.123194    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.123679 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.123660    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.131073 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.131051    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcrdm\" (UniqueName: \"kubernetes.io/projected/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-kube-api-access-rcrdm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.312037 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.311944    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:00.467375 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:00.467249    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"]
Jul 15 11:45:00.470204 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:45:00.470165    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3369948c_a8bf_4463_979e_e0bb7c8d2fa7.slice/crio-70aaab2b6d6829ace6c17605fcbc8bded9d1a262df5915939d2f1c2e235a7562 WatchSource:0}: Error finding container 70aaab2b6d6829ace6c17605fcbc8bded9d1a262df5915939d2f1c2e235a7562: Status 404 returned error can't find the container with id 70aaab2b6d6829ace6c17605fcbc8bded9d1a262df5915939d2f1c2e235a7562
Jul 15 11:45:01.286220 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:01.286188    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w" event={"ID":"3369948c-a8bf-4463-979e-e0bb7c8d2fa7","Type":"ContainerStarted","Data":"06efeae3745882639ffe405d6c94ce4cbe78abb7001f0aa99ffaa5d519e0a140"}
Jul 15 11:45:01.286624 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:01.286234    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w" event={"ID":"3369948c-a8bf-4463-979e-e0bb7c8d2fa7","Type":"ContainerStarted","Data":"70aaab2b6d6829ace6c17605fcbc8bded9d1a262df5915939d2f1c2e235a7562"}
Jul 15 11:45:01.305944 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:01.305898    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w" podStartSLOduration=2.305882615 podStartE2EDuration="2.305882615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:45:01.305271303 +0000 UTC m=+3570.535238837" watchObservedRunningTime="2026-07-15 11:45:01.305882615 +0000 UTC m=+3570.535850146"
Jul 15 11:45:01.312331 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:01.312311    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:10.312589 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:10.312539    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:10.332138 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:10.332103    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:10.343666 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:10.343640    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:33.805420 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:33.805370    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"]
Jul 15 11:45:38.489588 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:38.489550    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:45:38.492231 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:38.492207    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:45:48.826021 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:48.825974    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w" podUID="3369948c-a8bf-4463-979e-e0bb7c8d2fa7" containerName="main" containerID="cri-o://06efeae3745882639ffe405d6c94ce4cbe78abb7001f0aa99ffaa5d519e0a140" gracePeriod=45
Jul 15 11:45:49.085849 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.085794    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:49.147419 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.147380    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-dshm\") pod \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") "
Jul 15 11:45:49.147647 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.147443    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-tls-certs\") pod \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") "
Jul 15 11:45:49.147647 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.147475    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcrdm\" (UniqueName: \"kubernetes.io/projected/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-kube-api-access-rcrdm\") pod \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") "
Jul 15 11:45:49.147647 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.147521    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-model-cache\") pod \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") "
Jul 15 11:45:49.147647 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.147566    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-home\") pod \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") "
Jul 15 11:45:49.147890 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.147677    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-tmp-dir\") pod \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\" (UID: \"3369948c-a8bf-4463-979e-e0bb7c8d2fa7\") "
Jul 15 11:45:49.147890 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.147826    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-home" (OuterVolumeSpecName: "home") pod "3369948c-a8bf-4463-979e-e0bb7c8d2fa7" (UID: "3369948c-a8bf-4463-979e-e0bb7c8d2fa7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:45:49.147890 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.147843    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-model-cache" (OuterVolumeSpecName: "model-cache") pod "3369948c-a8bf-4463-979e-e0bb7c8d2fa7" (UID: "3369948c-a8bf-4463-979e-e0bb7c8d2fa7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:45:49.148046 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.147977    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:45:49.148046 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.147994    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:45:49.148319 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.148079    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3369948c-a8bf-4463-979e-e0bb7c8d2fa7" (UID: "3369948c-a8bf-4463-979e-e0bb7c8d2fa7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:45:49.149759 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.149722    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-dshm" (OuterVolumeSpecName: "dshm") pod "3369948c-a8bf-4463-979e-e0bb7c8d2fa7" (UID: "3369948c-a8bf-4463-979e-e0bb7c8d2fa7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:45:49.149875 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.149772    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3369948c-a8bf-4463-979e-e0bb7c8d2fa7" (UID: "3369948c-a8bf-4463-979e-e0bb7c8d2fa7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:45:49.149875 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.149843    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-kube-api-access-rcrdm" (OuterVolumeSpecName: "kube-api-access-rcrdm") pod "3369948c-a8bf-4463-979e-e0bb7c8d2fa7" (UID: "3369948c-a8bf-4463-979e-e0bb7c8d2fa7"). InnerVolumeSpecName "kube-api-access-rcrdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:45:49.248609 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.248566    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:45:49.248609 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.248609    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:45:49.248836 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.248628    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcrdm\" (UniqueName: \"kubernetes.io/projected/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-kube-api-access-rcrdm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:45:49.248836 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.248643    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3369948c-a8bf-4463-979e-e0bb7c8d2fa7-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:45:49.514015 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.513982    2566 generic.go:356] "Generic (PLEG): container finished" podID="3369948c-a8bf-4463-979e-e0bb7c8d2fa7" containerID="06efeae3745882639ffe405d6c94ce4cbe78abb7001f0aa99ffaa5d519e0a140" exitCode=0
Jul 15 11:45:49.514199 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.514027    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"
Jul 15 11:45:49.514199 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.514046    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w" event={"ID":"3369948c-a8bf-4463-979e-e0bb7c8d2fa7","Type":"ContainerDied","Data":"06efeae3745882639ffe405d6c94ce4cbe78abb7001f0aa99ffaa5d519e0a140"}
Jul 15 11:45:49.514199 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.514122    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w" event={"ID":"3369948c-a8bf-4463-979e-e0bb7c8d2fa7","Type":"ContainerDied","Data":"70aaab2b6d6829ace6c17605fcbc8bded9d1a262df5915939d2f1c2e235a7562"}
Jul 15 11:45:49.514199 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.514147    2566 scope.go:117] "RemoveContainer" containerID="06efeae3745882639ffe405d6c94ce4cbe78abb7001f0aa99ffaa5d519e0a140"
Jul 15 11:45:49.522456 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.522426    2566 scope.go:117] "RemoveContainer" containerID="06efeae3745882639ffe405d6c94ce4cbe78abb7001f0aa99ffaa5d519e0a140"
Jul 15 11:45:49.522722 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:45:49.522703    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06efeae3745882639ffe405d6c94ce4cbe78abb7001f0aa99ffaa5d519e0a140\": container with ID starting with 06efeae3745882639ffe405d6c94ce4cbe78abb7001f0aa99ffaa5d519e0a140 not found: ID does not exist" containerID="06efeae3745882639ffe405d6c94ce4cbe78abb7001f0aa99ffaa5d519e0a140"
Jul 15 11:45:49.522802 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.522729    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06efeae3745882639ffe405d6c94ce4cbe78abb7001f0aa99ffaa5d519e0a140"} err="failed to get container status \"06efeae3745882639ffe405d6c94ce4cbe78abb7001f0aa99ffaa5d519e0a140\": rpc error: code = NotFound desc = could not find container \"06efeae3745882639ffe405d6c94ce4cbe78abb7001f0aa99ffaa5d519e0a140\": container with ID starting with 06efeae3745882639ffe405d6c94ce4cbe78abb7001f0aa99ffaa5d519e0a140 not found: ID does not exist"
Jul 15 11:45:49.538751 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.538713    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"]
Jul 15 11:45:49.540721 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:49.540698    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8jq6w"]
Jul 15 11:45:51.269727 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:51.269684    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3369948c-a8bf-4463-979e-e0bb7c8d2fa7" path="/var/lib/kubelet/pods/3369948c-a8bf-4463-979e-e0bb7c8d2fa7/volumes"
Jul 15 11:45:56.728403 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.728371    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"]
Jul 15 11:45:56.728877 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.728701    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3369948c-a8bf-4463-979e-e0bb7c8d2fa7" containerName="main"
Jul 15 11:45:56.728877 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.728713    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3369948c-a8bf-4463-979e-e0bb7c8d2fa7" containerName="main"
Jul 15 11:45:56.728877 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.728787    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3369948c-a8bf-4463-979e-e0bb7c8d2fa7" containerName="main"
Jul 15 11:45:56.733215 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.733193    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.736010 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.735989    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-6rfrt\""
Jul 15 11:45:56.736871 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.736850    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 11:45:56.737016 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.736871    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 15 11:45:56.748790 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.748769    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"]
Jul 15 11:45:56.821902 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.821871    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.822059 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.821910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac7b7d8-6bda-4892-b80d-60072905bd35-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.822059 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.821944    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.822059 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.822009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs4s4\" (UniqueName: \"kubernetes.io/projected/cac7b7d8-6bda-4892-b80d-60072905bd35-kube-api-access-vs4s4\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.822059 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.822056    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.822199 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.822075    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-home\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.923455 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.923417    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-home\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.923655 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.923466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.923655 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.923519    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac7b7d8-6bda-4892-b80d-60072905bd35-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.923655 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.923569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.923655 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.923599    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs4s4\" (UniqueName: \"kubernetes.io/projected/cac7b7d8-6bda-4892-b80d-60072905bd35-kube-api-access-vs4s4\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.923655 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.923629    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.923940 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.923911    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-home\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.924069 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.924040    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.924147 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.924107    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.926051 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.926023    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.926244 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.926222    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac7b7d8-6bda-4892-b80d-60072905bd35-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.934321 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.934297    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs4s4\" (UniqueName: \"kubernetes.io/projected/cac7b7d8-6bda-4892-b80d-60072905bd35-kube-api-access-vs4s4\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:56.989189 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.989102    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"]
Jul 15 11:45:56.992657 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.992633    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"
Jul 15 11:45:56.995303 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:56.995277    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-f6v94\""
Jul 15 11:45:57.004762 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.004734    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"]
Jul 15 11:45:57.024314 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.024277    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ca3ff42e-24c8-4ed2-a39e-13ec28320765-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj\" (UID: \"ca3ff42e-24c8-4ed2-a39e-13ec28320765\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"
Jul 15 11:45:57.024630 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.024603    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcbq4\" (UniqueName: \"kubernetes.io/projected/ca3ff42e-24c8-4ed2-a39e-13ec28320765-kube-api-access-kcbq4\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj\" (UID: \"ca3ff42e-24c8-4ed2-a39e-13ec28320765\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"
Jul 15 11:45:57.024820 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.024785    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3ff42e-24c8-4ed2-a39e-13ec28320765-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj\" (UID: \"ca3ff42e-24c8-4ed2-a39e-13ec28320765\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"
Jul 15 11:45:57.041843 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.041819    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:57.126075 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.125930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcbq4\" (UniqueName: \"kubernetes.io/projected/ca3ff42e-24c8-4ed2-a39e-13ec28320765-kube-api-access-kcbq4\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj\" (UID: \"ca3ff42e-24c8-4ed2-a39e-13ec28320765\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"
Jul 15 11:45:57.126075 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.125981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3ff42e-24c8-4ed2-a39e-13ec28320765-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj\" (UID: \"ca3ff42e-24c8-4ed2-a39e-13ec28320765\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"
Jul 15 11:45:57.126075 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.126045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ca3ff42e-24c8-4ed2-a39e-13ec28320765-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj\" (UID: \"ca3ff42e-24c8-4ed2-a39e-13ec28320765\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"
Jul 15 11:45:57.126577 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.126481    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ca3ff42e-24c8-4ed2-a39e-13ec28320765-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj\" (UID: \"ca3ff42e-24c8-4ed2-a39e-13ec28320765\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"
Jul 15 11:45:57.129294 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.129267    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3ff42e-24c8-4ed2-a39e-13ec28320765-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj\" (UID: \"ca3ff42e-24c8-4ed2-a39e-13ec28320765\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"
Jul 15 11:45:57.135476 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.135447    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcbq4\" (UniqueName: \"kubernetes.io/projected/ca3ff42e-24c8-4ed2-a39e-13ec28320765-kube-api-access-kcbq4\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj\" (UID: \"ca3ff42e-24c8-4ed2-a39e-13ec28320765\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"
Jul 15 11:45:57.190755 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.190720    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"]
Jul 15 11:45:57.191886 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:45:57.191861    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac7b7d8_6bda_4892_b80d_60072905bd35.slice/crio-4ffee68b32b6602cbf3d18150dd5cecb8c0432f789798c3bd99c15c80a1df429 WatchSource:0}: Error finding container 4ffee68b32b6602cbf3d18150dd5cecb8c0432f789798c3bd99c15c80a1df429: Status 404 returned error can't find the container with id 4ffee68b32b6602cbf3d18150dd5cecb8c0432f789798c3bd99c15c80a1df429
Jul 15 11:45:57.307073 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.307040    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"
Jul 15 11:45:57.447276 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.447247    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"]
Jul 15 11:45:57.449907 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:45:57.449878    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3ff42e_24c8_4ed2_a39e_13ec28320765.slice/crio-48e7c2b80948bf612a3415f2d9247b5f66d8d7a58c83eec21683495d08daaf46 WatchSource:0}: Error finding container 48e7c2b80948bf612a3415f2d9247b5f66d8d7a58c83eec21683495d08daaf46: Status 404 returned error can't find the container with id 48e7c2b80948bf612a3415f2d9247b5f66d8d7a58c83eec21683495d08daaf46
Jul 15 11:45:57.554681 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.554641    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj" event={"ID":"ca3ff42e-24c8-4ed2-a39e-13ec28320765","Type":"ContainerStarted","Data":"571c686c0eec950083b92c1cac3fd801397c8b346ae8a27ab0442d56edd137da"}
Jul 15 11:45:57.554866 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.554694    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj" event={"ID":"ca3ff42e-24c8-4ed2-a39e-13ec28320765","Type":"ContainerStarted","Data":"48e7c2b80948bf612a3415f2d9247b5f66d8d7a58c83eec21683495d08daaf46"}
Jul 15 11:45:57.554866 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.554724    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"
Jul 15 11:45:57.556402 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.556372    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c" event={"ID":"cac7b7d8-6bda-4892-b80d-60072905bd35","Type":"ContainerStarted","Data":"d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec"}
Jul 15 11:45:57.556539 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.556427    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c" event={"ID":"cac7b7d8-6bda-4892-b80d-60072905bd35","Type":"ContainerStarted","Data":"4ffee68b32b6602cbf3d18150dd5cecb8c0432f789798c3bd99c15c80a1df429"}
Jul 15 11:45:57.556879 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.556856    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:45:57.576731 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:57.576679    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj" podStartSLOduration=1.576658309 podStartE2EDuration="1.576658309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:45:57.574386424 +0000 UTC m=+3626.804353956" watchObservedRunningTime="2026-07-15 11:45:57.576658309 +0000 UTC m=+3626.806625841"
Jul 15 11:45:58.562653 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:58.562617    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c" event={"ID":"cac7b7d8-6bda-4892-b80d-60072905bd35","Type":"ContainerStarted","Data":"437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690"}
Jul 15 11:45:58.586038 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:58.585991    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c" podStartSLOduration=2.585974927 podStartE2EDuration="2.585974927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:45:58.583921452 +0000 UTC m=+3627.813888993" watchObservedRunningTime="2026-07-15 11:45:58.585974927 +0000 UTC m=+3627.815942457"
Jul 15 11:45:59.042561 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:45:59.042525    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:46:07.042384 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:46:07.042347    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:46:07.056175 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:46:07.056150    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:46:07.056317 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:46:07.056288    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:46:07.614253 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:46:07.614219    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:46:28.564810 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:46:28.564780    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"
Jul 15 11:50:38.522783 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:50:38.522744    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:50:38.526198 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:50:38.526174    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:54:34.420264 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:34.420181    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"]
Jul 15 11:54:34.421822 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:34.421791    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"]
Jul 15 11:54:38.564394 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:38.564359    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj" podUID="ca3ff42e-24c8-4ed2-a39e-13ec28320765" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:54:48.564702 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:48.564658    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj" podUID="ca3ff42e-24c8-4ed2-a39e-13ec28320765" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:54:49.441151 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.441108    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj" podUID="ca3ff42e-24c8-4ed2-a39e-13ec28320765" containerName="main" containerID="cri-o://571c686c0eec950083b92c1cac3fd801397c8b346ae8a27ab0442d56edd137da" gracePeriod=45
Jul 15 11:54:49.442036 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.441421    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c" podUID="cac7b7d8-6bda-4892-b80d-60072905bd35" containerName="main" containerID="cri-o://437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690" gracePeriod=45
Jul 15 11:54:49.558959 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.558927    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c" podUID="cac7b7d8-6bda-4892-b80d-60072905bd35" containerName="llm-d-routing-sidecar" containerID="cri-o://d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec" gracePeriod=45
Jul 15 11:54:49.604104 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.604070    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c" podUID="cac7b7d8-6bda-4892-b80d-60072905bd35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Jul 15 11:54:49.701127 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.701058    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"
Jul 15 11:54:49.741715 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.741688    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ca3ff42e-24c8-4ed2-a39e-13ec28320765-tokenizer-uds\") pod \"ca3ff42e-24c8-4ed2-a39e-13ec28320765\" (UID: \"ca3ff42e-24c8-4ed2-a39e-13ec28320765\") "
Jul 15 11:54:49.741868 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.741765    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3ff42e-24c8-4ed2-a39e-13ec28320765-tls-certs\") pod \"ca3ff42e-24c8-4ed2-a39e-13ec28320765\" (UID: \"ca3ff42e-24c8-4ed2-a39e-13ec28320765\") "
Jul 15 11:54:49.741868 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.741817    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcbq4\" (UniqueName: \"kubernetes.io/projected/ca3ff42e-24c8-4ed2-a39e-13ec28320765-kube-api-access-kcbq4\") pod \"ca3ff42e-24c8-4ed2-a39e-13ec28320765\" (UID: \"ca3ff42e-24c8-4ed2-a39e-13ec28320765\") "
Jul 15 11:54:49.742029 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.741997    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca3ff42e-24c8-4ed2-a39e-13ec28320765-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "ca3ff42e-24c8-4ed2-a39e-13ec28320765" (UID: "ca3ff42e-24c8-4ed2-a39e-13ec28320765"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:54:49.745166 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.745139    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca3ff42e-24c8-4ed2-a39e-13ec28320765-kube-api-access-kcbq4" (OuterVolumeSpecName: "kube-api-access-kcbq4") pod "ca3ff42e-24c8-4ed2-a39e-13ec28320765" (UID: "ca3ff42e-24c8-4ed2-a39e-13ec28320765"). InnerVolumeSpecName "kube-api-access-kcbq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:54:49.746511 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.746475    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca3ff42e-24c8-4ed2-a39e-13ec28320765-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca3ff42e-24c8-4ed2-a39e-13ec28320765" (UID: "ca3ff42e-24c8-4ed2-a39e-13ec28320765"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:54:49.801945 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.801921    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:54:49.842810 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.842778    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs4s4\" (UniqueName: \"kubernetes.io/projected/cac7b7d8-6bda-4892-b80d-60072905bd35-kube-api-access-vs4s4\") pod \"cac7b7d8-6bda-4892-b80d-60072905bd35\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") "
Jul 15 11:54:49.842963 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.842831    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-home\") pod \"cac7b7d8-6bda-4892-b80d-60072905bd35\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") "
Jul 15 11:54:49.842963 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.842878    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-model-cache\") pod \"cac7b7d8-6bda-4892-b80d-60072905bd35\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") "
Jul 15 11:54:49.842963 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.842910    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-tmp-dir\") pod \"cac7b7d8-6bda-4892-b80d-60072905bd35\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") "
Jul 15 11:54:49.842963 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.842950    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-dshm\") pod \"cac7b7d8-6bda-4892-b80d-60072905bd35\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") "
Jul 15 11:54:49.843185 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.843041    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac7b7d8-6bda-4892-b80d-60072905bd35-tls-certs\") pod \"cac7b7d8-6bda-4892-b80d-60072905bd35\" (UID: \"cac7b7d8-6bda-4892-b80d-60072905bd35\") "
Jul 15 11:54:49.843185 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.843093    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-home" (OuterVolumeSpecName: "home") pod "cac7b7d8-6bda-4892-b80d-60072905bd35" (UID: "cac7b7d8-6bda-4892-b80d-60072905bd35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:54:49.843185 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.843113    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-model-cache" (OuterVolumeSpecName: "model-cache") pod "cac7b7d8-6bda-4892-b80d-60072905bd35" (UID: "cac7b7d8-6bda-4892-b80d-60072905bd35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:54:49.843333 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.843236    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cac7b7d8-6bda-4892-b80d-60072905bd35" (UID: "cac7b7d8-6bda-4892-b80d-60072905bd35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:54:49.843397 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.843316    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ca3ff42e-24c8-4ed2-a39e-13ec28320765-tokenizer-uds\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:54:49.843397 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.843348    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3ff42e-24c8-4ed2-a39e-13ec28320765-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:54:49.843397 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.843361    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:54:49.843397 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.843375    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:54:49.843574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.843408    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:54:49.843574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.843424    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcbq4\" (UniqueName: \"kubernetes.io/projected/ca3ff42e-24c8-4ed2-a39e-13ec28320765-kube-api-access-kcbq4\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:54:49.845212 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.845188    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac7b7d8-6bda-4892-b80d-60072905bd35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cac7b7d8-6bda-4892-b80d-60072905bd35" (UID: "cac7b7d8-6bda-4892-b80d-60072905bd35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:54:49.845617 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.845595    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-dshm" (OuterVolumeSpecName: "dshm") pod "cac7b7d8-6bda-4892-b80d-60072905bd35" (UID: "cac7b7d8-6bda-4892-b80d-60072905bd35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:54:49.845617 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.845603    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac7b7d8-6bda-4892-b80d-60072905bd35-kube-api-access-vs4s4" (OuterVolumeSpecName: "kube-api-access-vs4s4") pod "cac7b7d8-6bda-4892-b80d-60072905bd35" (UID: "cac7b7d8-6bda-4892-b80d-60072905bd35"). InnerVolumeSpecName "kube-api-access-vs4s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:54:49.944100 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.944059    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cac7b7d8-6bda-4892-b80d-60072905bd35-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:54:49.944100 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.944093    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cac7b7d8-6bda-4892-b80d-60072905bd35-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:54:49.944100 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:49.944107    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs4s4\" (UniqueName: \"kubernetes.io/projected/cac7b7d8-6bda-4892-b80d-60072905bd35-kube-api-access-vs4s4\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:54:50.075207 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.075174    2566 generic.go:356] "Generic (PLEG): container finished" podID="cac7b7d8-6bda-4892-b80d-60072905bd35" containerID="437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690" exitCode=0
Jul 15 11:54:50.075207 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.075200    2566 generic.go:356] "Generic (PLEG): container finished" podID="cac7b7d8-6bda-4892-b80d-60072905bd35" containerID="d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec" exitCode=0
Jul 15 11:54:50.075470 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.075228    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"
Jul 15 11:54:50.075470 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.075253    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c" event={"ID":"cac7b7d8-6bda-4892-b80d-60072905bd35","Type":"ContainerDied","Data":"437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690"}
Jul 15 11:54:50.075470 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.075312    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c" event={"ID":"cac7b7d8-6bda-4892-b80d-60072905bd35","Type":"ContainerDied","Data":"d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec"}
Jul 15 11:54:50.075470 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.075338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c" event={"ID":"cac7b7d8-6bda-4892-b80d-60072905bd35","Type":"ContainerDied","Data":"4ffee68b32b6602cbf3d18150dd5cecb8c0432f789798c3bd99c15c80a1df429"}
Jul 15 11:54:50.075470 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.075367    2566 scope.go:117] "RemoveContainer" containerID="437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690"
Jul 15 11:54:50.076748 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.076715    2566 generic.go:356] "Generic (PLEG): container finished" podID="ca3ff42e-24c8-4ed2-a39e-13ec28320765" containerID="571c686c0eec950083b92c1cac3fd801397c8b346ae8a27ab0442d56edd137da" exitCode=0
Jul 15 11:54:50.077014 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.076766    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"
Jul 15 11:54:50.077014 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.076819    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj" event={"ID":"ca3ff42e-24c8-4ed2-a39e-13ec28320765","Type":"ContainerDied","Data":"571c686c0eec950083b92c1cac3fd801397c8b346ae8a27ab0442d56edd137da"}
Jul 15 11:54:50.077014 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.076903    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj" event={"ID":"ca3ff42e-24c8-4ed2-a39e-13ec28320765","Type":"ContainerDied","Data":"48e7c2b80948bf612a3415f2d9247b5f66d8d7a58c83eec21683495d08daaf46"}
Jul 15 11:54:50.087455 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.085533    2566 scope.go:117] "RemoveContainer" containerID="d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec"
Jul 15 11:54:50.095419 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.095397    2566 scope.go:117] "RemoveContainer" containerID="437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690"
Jul 15 11:54:50.095745 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:54:50.095724    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690\": container with ID starting with 437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690 not found: ID does not exist" containerID="437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690"
Jul 15 11:54:50.095807 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.095760    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690"} err="failed to get container status \"437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690\": rpc error: code = NotFound desc = could not find container \"437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690\": container with ID starting with 437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690 not found: ID does not exist"
Jul 15 11:54:50.095807 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.095781    2566 scope.go:117] "RemoveContainer" containerID="d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec"
Jul 15 11:54:50.096021 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:54:50.096003    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec\": container with ID starting with d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec not found: ID does not exist" containerID="d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec"
Jul 15 11:54:50.096075 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.096026    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec"} err="failed to get container status \"d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec\": rpc error: code = NotFound desc = could not find container \"d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec\": container with ID starting with d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec not found: ID does not exist"
Jul 15 11:54:50.096075 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.096041    2566 scope.go:117] "RemoveContainer" containerID="437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690"
Jul 15 11:54:50.096304 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.096282    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690"} err="failed to get container status \"437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690\": rpc error: code = NotFound desc = could not find container \"437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690\": container with ID starting with 437c3ac1ba85f23859b1f2aa5e5587a54ef5879673dcca3d66c6e099a9c29690 not found: ID does not exist"
Jul 15 11:54:50.096376 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.096307    2566 scope.go:117] "RemoveContainer" containerID="d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec"
Jul 15 11:54:50.096568 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.096544    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec"} err="failed to get container status \"d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec\": rpc error: code = NotFound desc = could not find container \"d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec\": container with ID starting with d62cb4f4f9c8d467d600e8b32a15871b152ed141d1de6dd6bfd54faa05d27aec not found: ID does not exist"
Jul 15 11:54:50.096619 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.096572    2566 scope.go:117] "RemoveContainer" containerID="571c686c0eec950083b92c1cac3fd801397c8b346ae8a27ab0442d56edd137da"
Jul 15 11:54:50.105525 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.105506    2566 scope.go:117] "RemoveContainer" containerID="571c686c0eec950083b92c1cac3fd801397c8b346ae8a27ab0442d56edd137da"
Jul 15 11:54:50.105792 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:54:50.105770    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571c686c0eec950083b92c1cac3fd801397c8b346ae8a27ab0442d56edd137da\": container with ID starting with 571c686c0eec950083b92c1cac3fd801397c8b346ae8a27ab0442d56edd137da not found: ID does not exist" containerID="571c686c0eec950083b92c1cac3fd801397c8b346ae8a27ab0442d56edd137da"
Jul 15 11:54:50.105865 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.105800    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571c686c0eec950083b92c1cac3fd801397c8b346ae8a27ab0442d56edd137da"} err="failed to get container status \"571c686c0eec950083b92c1cac3fd801397c8b346ae8a27ab0442d56edd137da\": rpc error: code = NotFound desc = could not find container \"571c686c0eec950083b92c1cac3fd801397c8b346ae8a27ab0442d56edd137da\": container with ID starting with 571c686c0eec950083b92c1cac3fd801397c8b346ae8a27ab0442d56edd137da not found: ID does not exist"
Jul 15 11:54:50.110215 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.110196    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"]
Jul 15 11:54:50.113808 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.113790    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6fsn5qj"]
Jul 15 11:54:50.139818 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.139789    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"]
Jul 15 11:54:50.144579 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:50.144553    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-znk9c"]
Jul 15 11:54:51.268997 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:51.268949    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca3ff42e-24c8-4ed2-a39e-13ec28320765" path="/var/lib/kubelet/pods/ca3ff42e-24c8-4ed2-a39e-13ec28320765/volumes"
Jul 15 11:54:51.271640 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:54:51.269542    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cac7b7d8-6bda-4892-b80d-60072905bd35" path="/var/lib/kubelet/pods/cac7b7d8-6bda-4892-b80d-60072905bd35/volumes"
Jul 15 11:55:03.966656 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:03.966627    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"]
Jul 15 11:55:03.967043 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:03.966907    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca3ff42e-24c8-4ed2-a39e-13ec28320765" containerName="main"
Jul 15 11:55:03.967043 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:03.966917    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca3ff42e-24c8-4ed2-a39e-13ec28320765" containerName="main"
Jul 15 11:55:03.967043 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:03.966929    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cac7b7d8-6bda-4892-b80d-60072905bd35" containerName="main"
Jul 15 11:55:03.967043 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:03.966934    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cac7b7d8-6bda-4892-b80d-60072905bd35" containerName="main"
Jul 15 11:55:03.967043 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:03.966943    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cac7b7d8-6bda-4892-b80d-60072905bd35" containerName="llm-d-routing-sidecar"
Jul 15 11:55:03.967043 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:03.966949    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cac7b7d8-6bda-4892-b80d-60072905bd35" containerName="llm-d-routing-sidecar"
Jul 15 11:55:03.967043 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:03.966999    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cac7b7d8-6bda-4892-b80d-60072905bd35" containerName="main"
Jul 15 11:55:03.967043 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:03.967006    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cac7b7d8-6bda-4892-b80d-60072905bd35" containerName="llm-d-routing-sidecar"
Jul 15 11:55:03.967043 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:03.967014    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca3ff42e-24c8-4ed2-a39e-13ec28320765" containerName="main"
Jul 15 11:55:03.970238 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:03.970215    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:03.976201 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:03.976174    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 11:55:03.976334 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:03.976243    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-v9mnk\""
Jul 15 11:55:03.982519 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:03.978195    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 15 11:55:03.994955 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:03.994930    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"]
Jul 15 11:55:04.064055 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.064013    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.064055 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.064061    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.064315 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.064085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzqmd\" (UniqueName: \"kubernetes.io/projected/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-kube-api-access-pzqmd\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.064315 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.064147    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.064315 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.064195    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.064315 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.064224    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-home\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.164912 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.164874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.165109 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.164922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.165109 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.165050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqmd\" (UniqueName: \"kubernetes.io/projected/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-kube-api-access-pzqmd\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.165223 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.165110    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.165223 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.165175    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.165223 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.165211    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-home\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.165388 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.165304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.165547 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.165527    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.165630 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.165592    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-home\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.167483 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.167454    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.167931 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.167907    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.174959 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.174932    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzqmd\" (UniqueName: \"kubernetes.io/projected/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-kube-api-access-pzqmd\") pod \"scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.259921 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.259887    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"]
Jul 15 11:55:04.263717 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.263690    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"
Jul 15 11:55:04.266590 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.266563    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-cbmbh\""
Jul 15 11:55:04.276046 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.276023    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"]
Jul 15 11:55:04.285649 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.285617    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:04.368023 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.367974    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched68fb4\" (UID: \"3727d2af-8cb6-492a-97f9-3d9b19c38ce1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"
Jul 15 11:55:04.368213 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.368113    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched68fb4\" (UID: \"3727d2af-8cb6-492a-97f9-3d9b19c38ce1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"
Jul 15 11:55:04.368270 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.368232    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62xj5\" (UniqueName: \"kubernetes.io/projected/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-kube-api-access-62xj5\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched68fb4\" (UID: \"3727d2af-8cb6-492a-97f9-3d9b19c38ce1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"
Jul 15 11:55:04.434858 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.434820    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"]
Jul 15 11:55:04.437885 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:55:04.437851    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56203d4_637d_48c2_b4d1_2d5b7aa3d487.slice/crio-968e37449ec70384430f6e6a9064d562b2c7d8090442b284e0393f87a270c485 WatchSource:0}: Error finding container 968e37449ec70384430f6e6a9064d562b2c7d8090442b284e0393f87a270c485: Status 404 returned error can't find the container with id 968e37449ec70384430f6e6a9064d562b2c7d8090442b284e0393f87a270c485
Jul 15 11:55:04.469219 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.469188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched68fb4\" (UID: \"3727d2af-8cb6-492a-97f9-3d9b19c38ce1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"
Jul 15 11:55:04.469368 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.469239    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62xj5\" (UniqueName: \"kubernetes.io/projected/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-kube-api-access-62xj5\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched68fb4\" (UID: \"3727d2af-8cb6-492a-97f9-3d9b19c38ce1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"
Jul 15 11:55:04.469368 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.469277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched68fb4\" (UID: \"3727d2af-8cb6-492a-97f9-3d9b19c38ce1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"
Jul 15 11:55:04.469631 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.469612    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched68fb4\" (UID: \"3727d2af-8cb6-492a-97f9-3d9b19c38ce1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"
Jul 15 11:55:04.471900 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.471879    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched68fb4\" (UID: \"3727d2af-8cb6-492a-97f9-3d9b19c38ce1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"
Jul 15 11:55:04.480103 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.480080    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62xj5\" (UniqueName: \"kubernetes.io/projected/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-kube-api-access-62xj5\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched68fb4\" (UID: \"3727d2af-8cb6-492a-97f9-3d9b19c38ce1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"
Jul 15 11:55:04.575651 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.575563    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"
Jul 15 11:55:04.711697 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:04.711673    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"]
Jul 15 11:55:04.713768 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:55:04.713744    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3727d2af_8cb6_492a_97f9_3d9b19c38ce1.slice/crio-ef40b2a1ecb99563d4451ff5d5e81c48ebb777eff0fa2cdbe69639228277a09c WatchSource:0}: Error finding container ef40b2a1ecb99563d4451ff5d5e81c48ebb777eff0fa2cdbe69639228277a09c: Status 404 returned error can't find the container with id ef40b2a1ecb99563d4451ff5d5e81c48ebb777eff0fa2cdbe69639228277a09c
Jul 15 11:55:05.151063 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:05.151027    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4" event={"ID":"3727d2af-8cb6-492a-97f9-3d9b19c38ce1","Type":"ContainerStarted","Data":"ebea87e0b3f4bf0f85fe8621665a94760c9871bb5c541803c2cd1d8c465f63ee"}
Jul 15 11:55:05.151580 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:05.151082    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4" event={"ID":"3727d2af-8cb6-492a-97f9-3d9b19c38ce1","Type":"ContainerStarted","Data":"ef40b2a1ecb99563d4451ff5d5e81c48ebb777eff0fa2cdbe69639228277a09c"}
Jul 15 11:55:05.151580 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:05.151104    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"
Jul 15 11:55:05.152537 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:05.152485    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5" event={"ID":"e56203d4-637d-48c2-b4d1-2d5b7aa3d487","Type":"ContainerStarted","Data":"c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd"}
Jul 15 11:55:05.152667 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:05.152559    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5" event={"ID":"e56203d4-637d-48c2-b4d1-2d5b7aa3d487","Type":"ContainerStarted","Data":"968e37449ec70384430f6e6a9064d562b2c7d8090442b284e0393f87a270c485"}
Jul 15 11:55:05.152721 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:05.152664    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:05.171866 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:05.171820    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4" podStartSLOduration=1.171804296 podStartE2EDuration="1.171804296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:55:05.170573802 +0000 UTC m=+4174.400541333" watchObservedRunningTime="2026-07-15 11:55:05.171804296 +0000 UTC m=+4174.401771827"
Jul 15 11:55:06.158459 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:06.158419    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5" event={"ID":"e56203d4-637d-48c2-b4d1-2d5b7aa3d487","Type":"ContainerStarted","Data":"e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1"}
Jul 15 11:55:06.181042 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:06.180979    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5" podStartSLOduration=3.180957143 podStartE2EDuration="3.180957143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:55:06.180525299 +0000 UTC m=+4175.410492827" watchObservedRunningTime="2026-07-15 11:55:06.180957143 +0000 UTC m=+4175.410924675"
Jul 15 11:55:06.286024 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:06.285987    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:14.286683 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:14.286633    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:14.306900 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:14.306865    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:14.308452 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:14.308431    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:15.211515 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:15.211469    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:55:36.160944 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:36.160916    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"
Jul 15 11:55:38.554190 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:38.554152    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:55:38.556952 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:55:38.556928    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 11:56:38.709588 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:38.709538    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"]
Jul 15 11:56:38.711198 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:38.711165    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"]
Jul 15 11:56:46.161030 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:46.160987    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4" podUID="3727d2af-8cb6-492a-97f9-3d9b19c38ce1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 11:56:53.732513 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:53.732423    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5" podUID="e56203d4-637d-48c2-b4d1-2d5b7aa3d487" containerName="main" containerID="cri-o://e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1" gracePeriod=45
Jul 15 11:56:53.734146 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:53.734088    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4" podUID="3727d2af-8cb6-492a-97f9-3d9b19c38ce1" containerName="main" containerID="cri-o://ebea87e0b3f4bf0f85fe8621665a94760c9871bb5c541803c2cd1d8c465f63ee" gracePeriod=45
Jul 15 11:56:53.850128 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:53.850060    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5" podUID="e56203d4-637d-48c2-b4d1-2d5b7aa3d487" containerName="llm-d-routing-sidecar" containerID="cri-o://c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd" gracePeriod=45
Jul 15 11:56:54.011509 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.011467    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"
Jul 15 11:56:54.086147 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.086123    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:56:54.140164 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.140133    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62xj5\" (UniqueName: \"kubernetes.io/projected/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-kube-api-access-62xj5\") pod \"3727d2af-8cb6-492a-97f9-3d9b19c38ce1\" (UID: \"3727d2af-8cb6-492a-97f9-3d9b19c38ce1\") "
Jul 15 11:56:54.140437 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.140206    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-tokenizer-uds\") pod \"3727d2af-8cb6-492a-97f9-3d9b19c38ce1\" (UID: \"3727d2af-8cb6-492a-97f9-3d9b19c38ce1\") "
Jul 15 11:56:54.140437 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.140260    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-tls-certs\") pod \"3727d2af-8cb6-492a-97f9-3d9b19c38ce1\" (UID: \"3727d2af-8cb6-492a-97f9-3d9b19c38ce1\") "
Jul 15 11:56:54.140574 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.140485    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3727d2af-8cb6-492a-97f9-3d9b19c38ce1" (UID: "3727d2af-8cb6-492a-97f9-3d9b19c38ce1"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:56:54.142440 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.142403    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3727d2af-8cb6-492a-97f9-3d9b19c38ce1" (UID: "3727d2af-8cb6-492a-97f9-3d9b19c38ce1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:56:54.142583 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.142519    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-kube-api-access-62xj5" (OuterVolumeSpecName: "kube-api-access-62xj5") pod "3727d2af-8cb6-492a-97f9-3d9b19c38ce1" (UID: "3727d2af-8cb6-492a-97f9-3d9b19c38ce1"). InnerVolumeSpecName "kube-api-access-62xj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:56:54.241105 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.241076    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-model-cache\") pod \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") "
Jul 15 11:56:54.241286 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.241113    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-dshm\") pod \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") "
Jul 15 11:56:54.241286 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.241132    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-tls-certs\") pod \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") "
Jul 15 11:56:54.241286 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.241149    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzqmd\" (UniqueName: \"kubernetes.io/projected/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-kube-api-access-pzqmd\") pod \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") "
Jul 15 11:56:54.241286 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.241223    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-tmp-dir\") pod \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") "
Jul 15 11:56:54.241286 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.241250    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-home\") pod \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\" (UID: \"e56203d4-637d-48c2-b4d1-2d5b7aa3d487\") "
Jul 15 11:56:54.241565 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.241385    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-model-cache" (OuterVolumeSpecName: "model-cache") pod "e56203d4-637d-48c2-b4d1-2d5b7aa3d487" (UID: "e56203d4-637d-48c2-b4d1-2d5b7aa3d487"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:56:54.241565 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.241533    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62xj5\" (UniqueName: \"kubernetes.io/projected/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-kube-api-access-62xj5\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:56:54.241565 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.241552    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-tokenizer-uds\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:56:54.241710 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.241567    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:56:54.241710 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.241580    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3727d2af-8cb6-492a-97f9-3d9b19c38ce1-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:56:54.241710 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.241573    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e56203d4-637d-48c2-b4d1-2d5b7aa3d487" (UID: "e56203d4-637d-48c2-b4d1-2d5b7aa3d487"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:56:54.241710 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.241597    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-home" (OuterVolumeSpecName: "home") pod "e56203d4-637d-48c2-b4d1-2d5b7aa3d487" (UID: "e56203d4-637d-48c2-b4d1-2d5b7aa3d487"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:56:54.243562 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.243538    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-kube-api-access-pzqmd" (OuterVolumeSpecName: "kube-api-access-pzqmd") pod "e56203d4-637d-48c2-b4d1-2d5b7aa3d487" (UID: "e56203d4-637d-48c2-b4d1-2d5b7aa3d487"). InnerVolumeSpecName "kube-api-access-pzqmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:56:54.243742 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.243724    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e56203d4-637d-48c2-b4d1-2d5b7aa3d487" (UID: "e56203d4-637d-48c2-b4d1-2d5b7aa3d487"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:56:54.243742 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.243726    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-dshm" (OuterVolumeSpecName: "dshm") pod "e56203d4-637d-48c2-b4d1-2d5b7aa3d487" (UID: "e56203d4-637d-48c2-b4d1-2d5b7aa3d487"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:56:54.343125 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.343084    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:56:54.343125 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.343123    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:56:54.343323 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.343138    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzqmd\" (UniqueName: \"kubernetes.io/projected/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-kube-api-access-pzqmd\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:56:54.343323 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.343151    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:56:54.343323 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.343164    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e56203d4-637d-48c2-b4d1-2d5b7aa3d487-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:56:54.666349 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.666257    2566 generic.go:356] "Generic (PLEG): container finished" podID="e56203d4-637d-48c2-b4d1-2d5b7aa3d487" containerID="e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1" exitCode=0
Jul 15 11:56:54.666349 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.666281    2566 generic.go:356] "Generic (PLEG): container finished" podID="e56203d4-637d-48c2-b4d1-2d5b7aa3d487" containerID="c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd" exitCode=0
Jul 15 11:56:54.666349 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.666314    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"
Jul 15 11:56:54.666667 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.666344    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5" event={"ID":"e56203d4-637d-48c2-b4d1-2d5b7aa3d487","Type":"ContainerDied","Data":"e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1"}
Jul 15 11:56:54.666667 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.666408    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5" event={"ID":"e56203d4-637d-48c2-b4d1-2d5b7aa3d487","Type":"ContainerDied","Data":"c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd"}
Jul 15 11:56:54.666667 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.666434    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5" event={"ID":"e56203d4-637d-48c2-b4d1-2d5b7aa3d487","Type":"ContainerDied","Data":"968e37449ec70384430f6e6a9064d562b2c7d8090442b284e0393f87a270c485"}
Jul 15 11:56:54.666667 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.666462    2566 scope.go:117] "RemoveContainer" containerID="e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1"
Jul 15 11:56:54.667756 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.667738    2566 generic.go:356] "Generic (PLEG): container finished" podID="3727d2af-8cb6-492a-97f9-3d9b19c38ce1" containerID="ebea87e0b3f4bf0f85fe8621665a94760c9871bb5c541803c2cd1d8c465f63ee" exitCode=0
Jul 15 11:56:54.667850 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.667797    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4" event={"ID":"3727d2af-8cb6-492a-97f9-3d9b19c38ce1","Type":"ContainerDied","Data":"ebea87e0b3f4bf0f85fe8621665a94760c9871bb5c541803c2cd1d8c465f63ee"}
Jul 15 11:56:54.667850 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.667819    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"
Jul 15 11:56:54.667850 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.667836    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4" event={"ID":"3727d2af-8cb6-492a-97f9-3d9b19c38ce1","Type":"ContainerDied","Data":"ef40b2a1ecb99563d4451ff5d5e81c48ebb777eff0fa2cdbe69639228277a09c"}
Jul 15 11:56:54.675988 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.675846    2566 scope.go:117] "RemoveContainer" containerID="c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd"
Jul 15 11:56:54.684612 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.684591    2566 scope.go:117] "RemoveContainer" containerID="e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1"
Jul 15 11:56:54.684870 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:56:54.684852    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1\": container with ID starting with e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1 not found: ID does not exist" containerID="e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1"
Jul 15 11:56:54.684920 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.684879    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1"} err="failed to get container status \"e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1\": rpc error: code = NotFound desc = could not find container \"e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1\": container with ID starting with e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1 not found: ID does not exist"
Jul 15 11:56:54.684920 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.684897    2566 scope.go:117] "RemoveContainer" containerID="c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd"
Jul 15 11:56:54.685132 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:56:54.685114    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd\": container with ID starting with c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd not found: ID does not exist" containerID="c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd"
Jul 15 11:56:54.685181 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.685136    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd"} err="failed to get container status \"c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd\": rpc error: code = NotFound desc = could not find container \"c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd\": container with ID starting with c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd not found: ID does not exist"
Jul 15 11:56:54.685181 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.685148    2566 scope.go:117] "RemoveContainer" containerID="e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1"
Jul 15 11:56:54.685366 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.685351    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1"} err="failed to get container status \"e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1\": rpc error: code = NotFound desc = could not find container \"e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1\": container with ID starting with e044092ec82067bf816187d819907a962f1fbf568c052c5cf89400ba2774cea1 not found: ID does not exist"
Jul 15 11:56:54.685410 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.685366    2566 scope.go:117] "RemoveContainer" containerID="c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd"
Jul 15 11:56:54.685654 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.685629    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd"} err="failed to get container status \"c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd\": rpc error: code = NotFound desc = could not find container \"c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd\": container with ID starting with c4364cc8d8457d01fda34a63f189e6e1abf664013775781f0669faafe4cab2bd not found: ID does not exist"
Jul 15 11:56:54.685712 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.685657    2566 scope.go:117] "RemoveContainer" containerID="ebea87e0b3f4bf0f85fe8621665a94760c9871bb5c541803c2cd1d8c465f63ee"
Jul 15 11:56:54.691406 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.691381    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"]
Jul 15 11:56:54.693790 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.693770    2566 scope.go:117] "RemoveContainer" containerID="ebea87e0b3f4bf0f85fe8621665a94760c9871bb5c541803c2cd1d8c465f63ee"
Jul 15 11:56:54.694133 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:56:54.694095    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebea87e0b3f4bf0f85fe8621665a94760c9871bb5c541803c2cd1d8c465f63ee\": container with ID starting with ebea87e0b3f4bf0f85fe8621665a94760c9871bb5c541803c2cd1d8c465f63ee not found: ID does not exist" containerID="ebea87e0b3f4bf0f85fe8621665a94760c9871bb5c541803c2cd1d8c465f63ee"
Jul 15 11:56:54.694266 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.694140    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebea87e0b3f4bf0f85fe8621665a94760c9871bb5c541803c2cd1d8c465f63ee"} err="failed to get container status \"ebea87e0b3f4bf0f85fe8621665a94760c9871bb5c541803c2cd1d8c465f63ee\": rpc error: code = NotFound desc = could not find container \"ebea87e0b3f4bf0f85fe8621665a94760c9871bb5c541803c2cd1d8c465f63ee\": container with ID starting with ebea87e0b3f4bf0f85fe8621665a94760c9871bb5c541803c2cd1d8c465f63ee not found: ID does not exist"
Jul 15 11:56:54.695765 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.695745    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-859798d76c-knzc5"]
Jul 15 11:56:54.710426 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.710405    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"]
Jul 15 11:56:54.714695 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:54.714675    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched68fb4"]
Jul 15 11:56:55.268452 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:55.268422    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3727d2af-8cb6-492a-97f9-3d9b19c38ce1" path="/var/lib/kubelet/pods/3727d2af-8cb6-492a-97f9-3d9b19c38ce1/volumes"
Jul 15 11:56:55.268985 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:56:55.268812    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e56203d4-637d-48c2-b4d1-2d5b7aa3d487" path="/var/lib/kubelet/pods/e56203d4-637d-48c2-b4d1-2d5b7aa3d487/volumes"
Jul 15 11:57:16.908613 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:16.908579    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"]
Jul 15 11:57:16.909113 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:16.908856    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e56203d4-637d-48c2-b4d1-2d5b7aa3d487" containerName="main"
Jul 15 11:57:16.909113 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:16.908867    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e56203d4-637d-48c2-b4d1-2d5b7aa3d487" containerName="main"
Jul 15 11:57:16.909113 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:16.908881    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e56203d4-637d-48c2-b4d1-2d5b7aa3d487" containerName="llm-d-routing-sidecar"
Jul 15 11:57:16.909113 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:16.908888    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e56203d4-637d-48c2-b4d1-2d5b7aa3d487" containerName="llm-d-routing-sidecar"
Jul 15 11:57:16.909113 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:16.908905    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3727d2af-8cb6-492a-97f9-3d9b19c38ce1" containerName="main"
Jul 15 11:57:16.909113 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:16.908910    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3727d2af-8cb6-492a-97f9-3d9b19c38ce1" containerName="main"
Jul 15 11:57:16.909113 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:16.908952    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e56203d4-637d-48c2-b4d1-2d5b7aa3d487" containerName="main"
Jul 15 11:57:16.909113 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:16.908958    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e56203d4-637d-48c2-b4d1-2d5b7aa3d487" containerName="llm-d-routing-sidecar"
Jul 15 11:57:16.909113 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:16.908964    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3727d2af-8cb6-492a-97f9-3d9b19c38ce1" containerName="main"
Jul 15 11:57:16.913265 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:16.913247    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:16.917511 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:16.917478    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 11:57:16.917612 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:16.917516    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 15 11:57:16.923265 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:16.923246    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"]
Jul 15 11:57:17.027120 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.027072    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.027120 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.027123    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvr9x\" (UniqueName: \"kubernetes.io/projected/b950105f-2ae8-49b6-be42-e24d8e838cfa-kube-api-access-rvr9x\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.027382 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.027149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b950105f-2ae8-49b6-be42-e24d8e838cfa-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.027382 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.027185    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.027382 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.027243    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.027382 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.027275    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.128339 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.128305    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.128339 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.128356    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.128645 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.128410    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.128645 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.128437    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvr9x\" (UniqueName: \"kubernetes.io/projected/b950105f-2ae8-49b6-be42-e24d8e838cfa-kube-api-access-rvr9x\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.128645 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.128558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b950105f-2ae8-49b6-be42-e24d8e838cfa-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.128645 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.128621    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.128869 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.128808    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.128869 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.128843    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.128969 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.128888    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.131040 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.131021    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.131365 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.131332    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b950105f-2ae8-49b6-be42-e24d8e838cfa-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.146749 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.146716    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvr9x\" (UniqueName: \"kubernetes.io/projected/b950105f-2ae8-49b6-be42-e24d8e838cfa-kube-api-access-rvr9x\") pod \"rolling-upgrade-test-kserve-54bc76f55b-zl5nw\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.223137 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.223043    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:17.361708 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.361683    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"]
Jul 15 11:57:17.364439 ip-10-0-133-26 kubenswrapper[2566]: W0715 11:57:17.364413    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb950105f_2ae8_49b6_be42_e24d8e838cfa.slice/crio-08029fca9300ca2bbf3f61565ed2401a207519b3693ba21695f28e8c23f57559 WatchSource:0}: Error finding container 08029fca9300ca2bbf3f61565ed2401a207519b3693ba21695f28e8c23f57559: Status 404 returned error can't find the container with id 08029fca9300ca2bbf3f61565ed2401a207519b3693ba21695f28e8c23f57559
Jul 15 11:57:17.778262 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.778222    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw" event={"ID":"b950105f-2ae8-49b6-be42-e24d8e838cfa","Type":"ContainerStarted","Data":"cab70dacb2fc20c3e022e233c5f5a004dee1e3263d61cbefd4143dce20782716"}
Jul 15 11:57:17.778471 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.778288    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw" event={"ID":"b950105f-2ae8-49b6-be42-e24d8e838cfa","Type":"ContainerStarted","Data":"08029fca9300ca2bbf3f61565ed2401a207519b3693ba21695f28e8c23f57559"}
Jul 15 11:57:17.803404 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:17.803356    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw" podStartSLOduration=1.8033370880000001 podStartE2EDuration="1.803337088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:57:17.802960582 +0000 UTC m=+4307.032928113" watchObservedRunningTime="2026-07-15 11:57:17.803337088 +0000 UTC m=+4307.033304621"
Jul 15 11:57:18.223741 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:18.223653    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:27.224099 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:27.224009    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:27.236467 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:27.236438    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:57:27.835455 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:57:27.835424    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:58:03.087183 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:03.087140    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"]
Jul 15 11:58:18.115526 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.115444    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw" podUID="b950105f-2ae8-49b6-be42-e24d8e838cfa" containerName="main" containerID="cri-o://cab70dacb2fc20c3e022e233c5f5a004dee1e3263d61cbefd4143dce20782716" gracePeriod=45
Jul 15 11:58:18.369702 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.369640    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:58:18.453812 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.453788    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b950105f-2ae8-49b6-be42-e24d8e838cfa-tls-certs\") pod \"b950105f-2ae8-49b6-be42-e24d8e838cfa\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") "
Jul 15 11:58:18.453985 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.453829    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-dshm\") pod \"b950105f-2ae8-49b6-be42-e24d8e838cfa\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") "
Jul 15 11:58:18.453985 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.453864    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvr9x\" (UniqueName: \"kubernetes.io/projected/b950105f-2ae8-49b6-be42-e24d8e838cfa-kube-api-access-rvr9x\") pod \"b950105f-2ae8-49b6-be42-e24d8e838cfa\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") "
Jul 15 11:58:18.453985 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.453905    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-model-cache\") pod \"b950105f-2ae8-49b6-be42-e24d8e838cfa\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") "
Jul 15 11:58:18.453985 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.453931    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-tmp-dir\") pod \"b950105f-2ae8-49b6-be42-e24d8e838cfa\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") "
Jul 15 11:58:18.453985 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.453947    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-home\") pod \"b950105f-2ae8-49b6-be42-e24d8e838cfa\" (UID: \"b950105f-2ae8-49b6-be42-e24d8e838cfa\") "
Jul 15 11:58:18.454259 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.454232    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-model-cache" (OuterVolumeSpecName: "model-cache") pod "b950105f-2ae8-49b6-be42-e24d8e838cfa" (UID: "b950105f-2ae8-49b6-be42-e24d8e838cfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:58:18.454259 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.454241    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-home" (OuterVolumeSpecName: "home") pod "b950105f-2ae8-49b6-be42-e24d8e838cfa" (UID: "b950105f-2ae8-49b6-be42-e24d8e838cfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:58:18.454342 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.454249    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b950105f-2ae8-49b6-be42-e24d8e838cfa" (UID: "b950105f-2ae8-49b6-be42-e24d8e838cfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:58:18.455971 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.455949    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b950105f-2ae8-49b6-be42-e24d8e838cfa-kube-api-access-rvr9x" (OuterVolumeSpecName: "kube-api-access-rvr9x") pod "b950105f-2ae8-49b6-be42-e24d8e838cfa" (UID: "b950105f-2ae8-49b6-be42-e24d8e838cfa"). InnerVolumeSpecName "kube-api-access-rvr9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 11:58:18.456074 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.456044    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-dshm" (OuterVolumeSpecName: "dshm") pod "b950105f-2ae8-49b6-be42-e24d8e838cfa" (UID: "b950105f-2ae8-49b6-be42-e24d8e838cfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 11:58:18.456074 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.456051    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b950105f-2ae8-49b6-be42-e24d8e838cfa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b950105f-2ae8-49b6-be42-e24d8e838cfa" (UID: "b950105f-2ae8-49b6-be42-e24d8e838cfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 11:58:18.555017 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.554973    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:58:18.555017 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.555016    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:58:18.555227 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.555030    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b950105f-2ae8-49b6-be42-e24d8e838cfa-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:58:18.555227 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.555044    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:58:18.555227 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.555057    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvr9x\" (UniqueName: \"kubernetes.io/projected/b950105f-2ae8-49b6-be42-e24d8e838cfa-kube-api-access-rvr9x\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:58:18.555227 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:18.555070    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b950105f-2ae8-49b6-be42-e24d8e838cfa-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 11:58:19.060707 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:19.060674    2566 generic.go:356] "Generic (PLEG): container finished" podID="b950105f-2ae8-49b6-be42-e24d8e838cfa" containerID="cab70dacb2fc20c3e022e233c5f5a004dee1e3263d61cbefd4143dce20782716" exitCode=0
Jul 15 11:58:19.060878 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:19.060719    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"
Jul 15 11:58:19.060878 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:19.060721    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw" event={"ID":"b950105f-2ae8-49b6-be42-e24d8e838cfa","Type":"ContainerDied","Data":"cab70dacb2fc20c3e022e233c5f5a004dee1e3263d61cbefd4143dce20782716"}
Jul 15 11:58:19.060878 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:19.060775    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw" event={"ID":"b950105f-2ae8-49b6-be42-e24d8e838cfa","Type":"ContainerDied","Data":"08029fca9300ca2bbf3f61565ed2401a207519b3693ba21695f28e8c23f57559"}
Jul 15 11:58:19.060878 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:19.060803    2566 scope.go:117] "RemoveContainer" containerID="cab70dacb2fc20c3e022e233c5f5a004dee1e3263d61cbefd4143dce20782716"
Jul 15 11:58:19.071726 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:19.071705    2566 scope.go:117] "RemoveContainer" containerID="cab70dacb2fc20c3e022e233c5f5a004dee1e3263d61cbefd4143dce20782716"
Jul 15 11:58:19.072066 ip-10-0-133-26 kubenswrapper[2566]: E0715 11:58:19.072046    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab70dacb2fc20c3e022e233c5f5a004dee1e3263d61cbefd4143dce20782716\": container with ID starting with cab70dacb2fc20c3e022e233c5f5a004dee1e3263d61cbefd4143dce20782716 not found: ID does not exist" containerID="cab70dacb2fc20c3e022e233c5f5a004dee1e3263d61cbefd4143dce20782716"
Jul 15 11:58:19.072157 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:19.072079    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab70dacb2fc20c3e022e233c5f5a004dee1e3263d61cbefd4143dce20782716"} err="failed to get container status \"cab70dacb2fc20c3e022e233c5f5a004dee1e3263d61cbefd4143dce20782716\": rpc error: code = NotFound desc = could not find container \"cab70dacb2fc20c3e022e233c5f5a004dee1e3263d61cbefd4143dce20782716\": container with ID starting with cab70dacb2fc20c3e022e233c5f5a004dee1e3263d61cbefd4143dce20782716 not found: ID does not exist"
Jul 15 11:58:19.082525 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:19.082480    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"]
Jul 15 11:58:19.088458 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:19.088436    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-zl5nw"]
Jul 15 11:58:19.267588 ip-10-0-133-26 kubenswrapper[2566]: I0715 11:58:19.267549    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b950105f-2ae8-49b6-be42-e24d8e838cfa" path="/var/lib/kubelet/pods/b950105f-2ae8-49b6-be42-e24d8e838cfa/volumes"
Jul 15 12:00:38.578396 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:00:38.578312    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 12:00:38.581489 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:00:38.581469    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 12:01:28.932197 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:28.932156    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-798df69954-4m6w9"]
Jul 15 12:01:28.932758 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:28.932621    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b950105f-2ae8-49b6-be42-e24d8e838cfa" containerName="main"
Jul 15 12:01:28.932758 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:28.932638    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b950105f-2ae8-49b6-be42-e24d8e838cfa" containerName="main"
Jul 15 12:01:28.932758 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:28.932753    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b950105f-2ae8-49b6-be42-e24d8e838cfa" containerName="main"
Jul 15 12:01:28.935868 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:28.935848    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-798df69954-4m6w9"
Jul 15 12:01:28.939149 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:28.939129    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-7xrpg\""
Jul 15 12:01:28.939270 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:28.939197    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 15 12:01:28.944823 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:28.944151    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-798df69954-4m6w9"]
Jul 15 12:01:29.084255 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:29.084216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/623186bb-08fc-42c1-92fa-4cc2989afccb-cert\") pod \"llmisvc-controller-manager-798df69954-4m6w9\" (UID: \"623186bb-08fc-42c1-92fa-4cc2989afccb\") " pod="kserve/llmisvc-controller-manager-798df69954-4m6w9"
Jul 15 12:01:29.084443 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:29.084282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4llbh\" (UniqueName: \"kubernetes.io/projected/623186bb-08fc-42c1-92fa-4cc2989afccb-kube-api-access-4llbh\") pod \"llmisvc-controller-manager-798df69954-4m6w9\" (UID: \"623186bb-08fc-42c1-92fa-4cc2989afccb\") " pod="kserve/llmisvc-controller-manager-798df69954-4m6w9"
Jul 15 12:01:29.184910 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:29.184810    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/623186bb-08fc-42c1-92fa-4cc2989afccb-cert\") pod \"llmisvc-controller-manager-798df69954-4m6w9\" (UID: \"623186bb-08fc-42c1-92fa-4cc2989afccb\") " pod="kserve/llmisvc-controller-manager-798df69954-4m6w9"
Jul 15 12:01:29.184910 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:29.184879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4llbh\" (UniqueName: \"kubernetes.io/projected/623186bb-08fc-42c1-92fa-4cc2989afccb-kube-api-access-4llbh\") pod \"llmisvc-controller-manager-798df69954-4m6w9\" (UID: \"623186bb-08fc-42c1-92fa-4cc2989afccb\") " pod="kserve/llmisvc-controller-manager-798df69954-4m6w9"
Jul 15 12:01:29.187366 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:29.187332    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/623186bb-08fc-42c1-92fa-4cc2989afccb-cert\") pod \"llmisvc-controller-manager-798df69954-4m6w9\" (UID: \"623186bb-08fc-42c1-92fa-4cc2989afccb\") " pod="kserve/llmisvc-controller-manager-798df69954-4m6w9"
Jul 15 12:01:29.194346 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:29.194310    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4llbh\" (UniqueName: \"kubernetes.io/projected/623186bb-08fc-42c1-92fa-4cc2989afccb-kube-api-access-4llbh\") pod \"llmisvc-controller-manager-798df69954-4m6w9\" (UID: \"623186bb-08fc-42c1-92fa-4cc2989afccb\") " pod="kserve/llmisvc-controller-manager-798df69954-4m6w9"
Jul 15 12:01:29.248268 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:29.248230    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-798df69954-4m6w9"
Jul 15 12:01:29.391320 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:29.391163    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-798df69954-4m6w9"]
Jul 15 12:01:29.393914 ip-10-0-133-26 kubenswrapper[2566]: W0715 12:01:29.393884    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod623186bb_08fc_42c1_92fa_4cc2989afccb.slice/crio-eb3cdbfd92616afc0ffde097034d5d9ef942a7dae77a32d311c545e1ac6c537e WatchSource:0}: Error finding container eb3cdbfd92616afc0ffde097034d5d9ef942a7dae77a32d311c545e1ac6c537e: Status 404 returned error can't find the container with id eb3cdbfd92616afc0ffde097034d5d9ef942a7dae77a32d311c545e1ac6c537e
Jul 15 12:01:29.395173 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:29.395153    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 15 12:01:29.930789 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:29.930755    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-798df69954-4m6w9" event={"ID":"623186bb-08fc-42c1-92fa-4cc2989afccb","Type":"ContainerStarted","Data":"eb3cdbfd92616afc0ffde097034d5d9ef942a7dae77a32d311c545e1ac6c537e"}
Jul 15 12:01:34.957276 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:34.957238    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-798df69954-4m6w9" event={"ID":"623186bb-08fc-42c1-92fa-4cc2989afccb","Type":"ContainerStarted","Data":"33565eda89f62ca8ae43d4f05003ecc912c8f4d78f55f750c242a09e28624c4b"}
Jul 15 12:01:34.957776 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:34.957311    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-798df69954-4m6w9"
Jul 15 12:01:34.979411 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:01:34.979360    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-798df69954-4m6w9" podStartSLOduration=2.183757077 podStartE2EDuration="6.979345004s" totalImagesPullingTime="4.795587927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:01:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:01:34.977633521 +0000 UTC m=+4564.207601053" watchObservedRunningTime="2026-07-15 12:01:34.979345004 +0000 UTC m=+4564.209312535"
Jul 15 12:02:00.242257 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.242175    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"]
Jul 15 12:02:00.245399 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.245382    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:02:00.249820 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.249798    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-p7tbn\""
Jul 15 12:02:00.250803 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.250783    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 15 12:02:00.250925 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.250777    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 15 12:02:00.250989 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.250920    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-epp-sa-dockercfg-kwxs7\""
Jul 15 12:02:00.262488 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.262465    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"]
Jul 15 12:02:00.329937 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.329899    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7\" (UID: \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:02:00.329937 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.329942    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7\" (UID: \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:02:00.330173 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.329962    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxjlq\" (UniqueName: \"kubernetes.io/projected/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-kube-api-access-kxjlq\") pod \"tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7\" (UID: \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:02:00.330173 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.330077    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7\" (UID: \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:02:00.430606 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.430577    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7\" (UID: \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:02:00.430787 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.430617    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7\" (UID: \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:02:00.430787 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.430639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxjlq\" (UniqueName: \"kubernetes.io/projected/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-kube-api-access-kxjlq\") pod \"tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7\" (UID: \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:02:00.430787 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.430692    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7\" (UID: \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:02:00.431008 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.430975    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7\" (UID: \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:02:00.431008 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.430998    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7\" (UID: \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:02:00.433029 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.433004    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7\" (UID: \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:02:00.442534 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.442489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxjlq\" (UniqueName: \"kubernetes.io/projected/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-kube-api-access-kxjlq\") pod \"tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7\" (UID: \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:02:00.554486 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.554444    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:02:00.696301 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:00.696275    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"]
Jul 15 12:02:00.697168 ip-10-0-133-26 kubenswrapper[2566]: W0715 12:02:00.697139    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6cd081_4674_49f3_abf2_6f1efc5de7ab.slice/crio-4ecc9b8c5675e05635c709fee41a0f72c772f2fce9d9ef4f0160133605493f7a WatchSource:0}: Error finding container 4ecc9b8c5675e05635c709fee41a0f72c772f2fce9d9ef4f0160133605493f7a: Status 404 returned error can't find the container with id 4ecc9b8c5675e05635c709fee41a0f72c772f2fce9d9ef4f0160133605493f7a
Jul 15 12:02:01.068373 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:01.068336    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7" event={"ID":"7c6cd081-4674-49f3-abf2-6f1efc5de7ab","Type":"ContainerStarted","Data":"1132d8b5d2ee66fa77fee0834a87abe99effceb61968ba705b67a94142e10380"}
Jul 15 12:02:01.068588 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:01.068398    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7" event={"ID":"7c6cd081-4674-49f3-abf2-6f1efc5de7ab","Type":"ContainerStarted","Data":"4ecc9b8c5675e05635c709fee41a0f72c772f2fce9d9ef4f0160133605493f7a"}
Jul 15 12:02:02.074321 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:02.074287    2566 generic.go:356] "Generic (PLEG): container finished" podID="7c6cd081-4674-49f3-abf2-6f1efc5de7ab" containerID="1132d8b5d2ee66fa77fee0834a87abe99effceb61968ba705b67a94142e10380" exitCode=0
Jul 15 12:02:02.074749 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:02.074327    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7" event={"ID":"7c6cd081-4674-49f3-abf2-6f1efc5de7ab","Type":"ContainerDied","Data":"1132d8b5d2ee66fa77fee0834a87abe99effceb61968ba705b67a94142e10380"}
Jul 15 12:02:03.080293 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:03.080260    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7" event={"ID":"7c6cd081-4674-49f3-abf2-6f1efc5de7ab","Type":"ContainerStarted","Data":"feb09d9f24068c9aaeecb7bd4e9d85360bc3b5858a726f6876db156a9a41b752"}
Jul 15 12:02:03.080719 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:03.080351    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:02:03.110860 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:03.110811    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7" podStartSLOduration=2.464076303 podStartE2EDuration="3.110794799s" totalImagesPullingTime="0s" totalInitContainerRuntime="646.718496ms" isStatefulPod=false podCreationTimestamp="2026-07-15 12:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:02:03.109724594 +0000 UTC m=+4592.339692126" watchObservedRunningTime="2026-07-15 12:02:03.110794799 +0000 UTC m=+4592.340762329"
Jul 15 12:02:05.965435 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:05.965406    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-798df69954-4m6w9"
Jul 15 12:02:34.086308 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:34.086281    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:02:43.709267 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.709232    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"]
Jul 15 12:02:43.712891 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.712874    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.716716 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.716695    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 15 12:02:43.732737 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.732711    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"]
Jul 15 12:02:43.799787 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.799750    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-kserve-provision-location\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.799970 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.799800    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v44wx\" (UniqueName: \"kubernetes.io/projected/a62613a0-2162-45c9-86fb-c5c8977842d8-kube-api-access-v44wx\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.799970 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.799817    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-model-cache\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.799970 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.799882    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-tmp-dir\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.799970 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.799911    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-dshm\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.799970 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.799948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-home\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.800137 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.799974    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62613a0-2162-45c9-86fb-c5c8977842d8-tls-certs\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.900769 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.900733    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-kserve-provision-location\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.900954 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.900786    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v44wx\" (UniqueName: \"kubernetes.io/projected/a62613a0-2162-45c9-86fb-c5c8977842d8-kube-api-access-v44wx\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.900998 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.900968    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-model-cache\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.901036 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.901006    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-tmp-dir\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.901079 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.901034    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-dshm\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.901116 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.901092    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-home\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.901151 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.901127    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62613a0-2162-45c9-86fb-c5c8977842d8-tls-certs\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.901151 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.901137    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-kserve-provision-location\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.901352 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.901325    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-model-cache\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.901427 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.901411    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-home\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.901490 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.901414    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-tmp-dir\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.903547 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.903528    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-dshm\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.903720 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.903704    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62613a0-2162-45c9-86fb-c5c8977842d8-tls-certs\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:43.913930 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:43.913906    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v44wx\" (UniqueName: \"kubernetes.io/projected/a62613a0-2162-45c9-86fb-c5c8977842d8-kube-api-access-v44wx\") pod \"prestop-hook-test-kserve-85ccc87949-2bshq\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:44.022974 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.022933    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:44.024974 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.024944    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"]
Jul 15 12:02:44.028094 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.028071    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:02:44.032165 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.032143    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-epp-sa-dockercfg-8pwkm\""
Jul 15 12:02:44.069010 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.068980    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"]
Jul 15 12:02:44.102687 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.102660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7cvx\" (UniqueName: \"kubernetes.io/projected/87acd944-f0a2-491c-8d52-fccb70099d0a-kube-api-access-k7cvx\") pod \"prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8\" (UID: \"87acd944-f0a2-491c-8d52-fccb70099d0a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:02:44.102838 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.102718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87acd944-f0a2-491c-8d52-fccb70099d0a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8\" (UID: \"87acd944-f0a2-491c-8d52-fccb70099d0a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:02:44.102838 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.102769    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87acd944-f0a2-491c-8d52-fccb70099d0a-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8\" (UID: \"87acd944-f0a2-491c-8d52-fccb70099d0a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:02:44.102915 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.102850    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/87acd944-f0a2-491c-8d52-fccb70099d0a-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8\" (UID: \"87acd944-f0a2-491c-8d52-fccb70099d0a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:02:44.204069 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.204032    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87acd944-f0a2-491c-8d52-fccb70099d0a-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8\" (UID: \"87acd944-f0a2-491c-8d52-fccb70099d0a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:02:44.204398 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.204081    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/87acd944-f0a2-491c-8d52-fccb70099d0a-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8\" (UID: \"87acd944-f0a2-491c-8d52-fccb70099d0a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:02:44.204398 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.204135    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7cvx\" (UniqueName: \"kubernetes.io/projected/87acd944-f0a2-491c-8d52-fccb70099d0a-kube-api-access-k7cvx\") pod \"prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8\" (UID: \"87acd944-f0a2-491c-8d52-fccb70099d0a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:02:44.204398 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.204197    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87acd944-f0a2-491c-8d52-fccb70099d0a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8\" (UID: \"87acd944-f0a2-491c-8d52-fccb70099d0a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:02:44.204590 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.204571    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/87acd944-f0a2-491c-8d52-fccb70099d0a-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8\" (UID: \"87acd944-f0a2-491c-8d52-fccb70099d0a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:02:44.204643 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.204583    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87acd944-f0a2-491c-8d52-fccb70099d0a-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8\" (UID: \"87acd944-f0a2-491c-8d52-fccb70099d0a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:02:44.206875 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.206851    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87acd944-f0a2-491c-8d52-fccb70099d0a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8\" (UID: \"87acd944-f0a2-491c-8d52-fccb70099d0a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:02:44.212082 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.212060    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"]
Jul 15 12:02:44.214840 ip-10-0-133-26 kubenswrapper[2566]: W0715 12:02:44.214812    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62613a0_2162_45c9_86fb_c5c8977842d8.slice/crio-35c5b40fc2c9a11391ff0e30368fe835a202d5c10279a8a1eb056aeeaa955fa5 WatchSource:0}: Error finding container 35c5b40fc2c9a11391ff0e30368fe835a202d5c10279a8a1eb056aeeaa955fa5: Status 404 returned error can't find the container with id 35c5b40fc2c9a11391ff0e30368fe835a202d5c10279a8a1eb056aeeaa955fa5
Jul 15 12:02:44.225714 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.225687    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7cvx\" (UniqueName: \"kubernetes.io/projected/87acd944-f0a2-491c-8d52-fccb70099d0a-kube-api-access-k7cvx\") pod \"prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8\" (UID: \"87acd944-f0a2-491c-8d52-fccb70099d0a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:02:44.252538 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.252487    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" event={"ID":"a62613a0-2162-45c9-86fb-c5c8977842d8","Type":"ContainerStarted","Data":"35c5b40fc2c9a11391ff0e30368fe835a202d5c10279a8a1eb056aeeaa955fa5"}
Jul 15 12:02:44.352660 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.352620    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:02:44.512146 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:44.512122    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"]
Jul 15 12:02:44.515280 ip-10-0-133-26 kubenswrapper[2566]: W0715 12:02:44.515256    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87acd944_f0a2_491c_8d52_fccb70099d0a.slice/crio-a577f4685c8fa4c6db812437b8de517d1beaf442654b7210aae9a79c12cb43ea WatchSource:0}: Error finding container a577f4685c8fa4c6db812437b8de517d1beaf442654b7210aae9a79c12cb43ea: Status 404 returned error can't find the container with id a577f4685c8fa4c6db812437b8de517d1beaf442654b7210aae9a79c12cb43ea
Jul 15 12:02:45.259205 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:45.259167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8" event={"ID":"87acd944-f0a2-491c-8d52-fccb70099d0a","Type":"ContainerStarted","Data":"92abe81900d7ab7254bd53e88f1c3d70793c12a94fc105156c04329b6873adcd"}
Jul 15 12:02:45.259685 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:45.259263    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8" event={"ID":"87acd944-f0a2-491c-8d52-fccb70099d0a","Type":"ContainerStarted","Data":"a577f4685c8fa4c6db812437b8de517d1beaf442654b7210aae9a79c12cb43ea"}
Jul 15 12:02:45.260675 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:45.260647    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" event={"ID":"a62613a0-2162-45c9-86fb-c5c8977842d8","Type":"ContainerStarted","Data":"2c56e49f8f9cc57fac7dd967e781ae4a922f559f6bb6204611038e18f0eae65d"}
Jul 15 12:02:46.266564 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:46.266525    2566 generic.go:356] "Generic (PLEG): container finished" podID="87acd944-f0a2-491c-8d52-fccb70099d0a" containerID="92abe81900d7ab7254bd53e88f1c3d70793c12a94fc105156c04329b6873adcd" exitCode=0
Jul 15 12:02:46.267057 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:46.266604    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8" event={"ID":"87acd944-f0a2-491c-8d52-fccb70099d0a","Type":"ContainerDied","Data":"92abe81900d7ab7254bd53e88f1c3d70793c12a94fc105156c04329b6873adcd"}
Jul 15 12:02:47.272214 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:47.272182    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8" event={"ID":"87acd944-f0a2-491c-8d52-fccb70099d0a","Type":"ContainerStarted","Data":"ca0a03568dc396ca9a5fba3d6018625cd29ccb3b50ec787fe8bc32547debfb80"}
Jul 15 12:02:47.272709 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:47.272297    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:02:47.273706 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:47.273686    2566 generic.go:356] "Generic (PLEG): container finished" podID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerID="2c56e49f8f9cc57fac7dd967e781ae4a922f559f6bb6204611038e18f0eae65d" exitCode=0
Jul 15 12:02:47.273789 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:47.273758    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" event={"ID":"a62613a0-2162-45c9-86fb-c5c8977842d8","Type":"ContainerDied","Data":"2c56e49f8f9cc57fac7dd967e781ae4a922f559f6bb6204611038e18f0eae65d"}
Jul 15 12:02:47.318138 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:47.318091    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8" podStartSLOduration=3.575994931 podStartE2EDuration="4.318076973s" totalImagesPullingTime="0s" totalInitContainerRuntime="742.082042ms" isStatefulPod=false podCreationTimestamp="2026-07-15 12:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:02:47.313007008 +0000 UTC m=+4636.542974545" watchObservedRunningTime="2026-07-15 12:02:47.318076973 +0000 UTC m=+4636.548044504"
Jul 15 12:02:48.281452 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:48.281409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" event={"ID":"a62613a0-2162-45c9-86fb-c5c8977842d8","Type":"ContainerStarted","Data":"075020b843f2fa2b3f4cb20906af8dfc9834ea1b1ac617ae258c34cbfb789bb7"}
Jul 15 12:02:48.309181 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:48.309123    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" podStartSLOduration=2.434153964 podStartE2EDuration="5.309105777s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.874951813s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:02:48.308991808 +0000 UTC m=+4637.538959338" watchObservedRunningTime="2026-07-15 12:02:48.309105777 +0000 UTC m=+4637.539073330"
Jul 15 12:02:49.023679 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:49.023640    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:54.023489 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:54.023451    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:02:54.024786 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:02:54.024758    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 15 12:03:04.023355 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:03:04.023311    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 15 12:03:14.024175 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:03:14.024133    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 15 12:03:18.283336 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:03:18.283307    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:03:24.023881 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:03:24.023789    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 15 12:03:34.023869 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:03:34.023825    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 15 12:03:44.023888 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:03:44.023845    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 15 12:03:54.024040 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:03:54.023990    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 15 12:04:04.024236 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:04.024190    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 15 12:04:05.145397 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:05.145361    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"]
Jul 15 12:04:14.024133 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:14.024086    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 15 12:04:14.085877 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:14.085837    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7" podUID="7c6cd081-4674-49f3-abf2-6f1efc5de7ab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 12:04:20.165570 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.165520    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7" podUID="7c6cd081-4674-49f3-abf2-6f1efc5de7ab" containerName="main" containerID="cri-o://feb09d9f24068c9aaeecb7bd4e9d85360bc3b5858a726f6876db156a9a41b752" gracePeriod=45
Jul 15 12:04:20.415651 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.415589    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:04:20.477033 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.476998    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-kserve-provision-location\") pod \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\" (UID: \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\") "
Jul 15 12:04:20.477514 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.477115    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-tokenizer-uds\") pod \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\" (UID: \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\") "
Jul 15 12:04:20.477514 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.477150    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxjlq\" (UniqueName: \"kubernetes.io/projected/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-kube-api-access-kxjlq\") pod \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\" (UID: \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\") "
Jul 15 12:04:20.477514 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.477172    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-tls-certs\") pod \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\" (UID: \"7c6cd081-4674-49f3-abf2-6f1efc5de7ab\") "
Jul 15 12:04:20.477514 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.477429    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "7c6cd081-4674-49f3-abf2-6f1efc5de7ab" (UID: "7c6cd081-4674-49f3-abf2-6f1efc5de7ab"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:04:20.477782 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.477759    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c6cd081-4674-49f3-abf2-6f1efc5de7ab" (UID: "7c6cd081-4674-49f3-abf2-6f1efc5de7ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:04:20.479392 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.479366    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-kube-api-access-kxjlq" (OuterVolumeSpecName: "kube-api-access-kxjlq") pod "7c6cd081-4674-49f3-abf2-6f1efc5de7ab" (UID: "7c6cd081-4674-49f3-abf2-6f1efc5de7ab"). InnerVolumeSpecName "kube-api-access-kxjlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 12:04:20.479392 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.479374    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c6cd081-4674-49f3-abf2-6f1efc5de7ab" (UID: "7c6cd081-4674-49f3-abf2-6f1efc5de7ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 12:04:20.578823 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.578779    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-tokenizer-uds\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 12:04:20.578823 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.578808    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxjlq\" (UniqueName: \"kubernetes.io/projected/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-kube-api-access-kxjlq\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 12:04:20.578823 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.578818    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 12:04:20.578823 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.578828    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c6cd081-4674-49f3-abf2-6f1efc5de7ab-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 12:04:20.729176 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.729082    2566 generic.go:356] "Generic (PLEG): container finished" podID="7c6cd081-4674-49f3-abf2-6f1efc5de7ab" containerID="feb09d9f24068c9aaeecb7bd4e9d85360bc3b5858a726f6876db156a9a41b752" exitCode=0
Jul 15 12:04:20.729176 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.729145    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"
Jul 15 12:04:20.729176 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.729162    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7" event={"ID":"7c6cd081-4674-49f3-abf2-6f1efc5de7ab","Type":"ContainerDied","Data":"feb09d9f24068c9aaeecb7bd4e9d85360bc3b5858a726f6876db156a9a41b752"}
Jul 15 12:04:20.729456 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.729226    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7" event={"ID":"7c6cd081-4674-49f3-abf2-6f1efc5de7ab","Type":"ContainerDied","Data":"4ecc9b8c5675e05635c709fee41a0f72c772f2fce9d9ef4f0160133605493f7a"}
Jul 15 12:04:20.729456 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.729256    2566 scope.go:117] "RemoveContainer" containerID="feb09d9f24068c9aaeecb7bd4e9d85360bc3b5858a726f6876db156a9a41b752"
Jul 15 12:04:20.737813 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.737797    2566 scope.go:117] "RemoveContainer" containerID="1132d8b5d2ee66fa77fee0834a87abe99effceb61968ba705b67a94142e10380"
Jul 15 12:04:20.748443 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.748425    2566 scope.go:117] "RemoveContainer" containerID="feb09d9f24068c9aaeecb7bd4e9d85360bc3b5858a726f6876db156a9a41b752"
Jul 15 12:04:20.748740 ip-10-0-133-26 kubenswrapper[2566]: E0715 12:04:20.748722    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb09d9f24068c9aaeecb7bd4e9d85360bc3b5858a726f6876db156a9a41b752\": container with ID starting with feb09d9f24068c9aaeecb7bd4e9d85360bc3b5858a726f6876db156a9a41b752 not found: ID does not exist" containerID="feb09d9f24068c9aaeecb7bd4e9d85360bc3b5858a726f6876db156a9a41b752"
Jul 15 12:04:20.748804 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.748749    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb09d9f24068c9aaeecb7bd4e9d85360bc3b5858a726f6876db156a9a41b752"} err="failed to get container status \"feb09d9f24068c9aaeecb7bd4e9d85360bc3b5858a726f6876db156a9a41b752\": rpc error: code = NotFound desc = could not find container \"feb09d9f24068c9aaeecb7bd4e9d85360bc3b5858a726f6876db156a9a41b752\": container with ID starting with feb09d9f24068c9aaeecb7bd4e9d85360bc3b5858a726f6876db156a9a41b752 not found: ID does not exist"
Jul 15 12:04:20.748804 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.748765    2566 scope.go:117] "RemoveContainer" containerID="1132d8b5d2ee66fa77fee0834a87abe99effceb61968ba705b67a94142e10380"
Jul 15 12:04:20.749020 ip-10-0-133-26 kubenswrapper[2566]: E0715 12:04:20.749000    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1132d8b5d2ee66fa77fee0834a87abe99effceb61968ba705b67a94142e10380\": container with ID starting with 1132d8b5d2ee66fa77fee0834a87abe99effceb61968ba705b67a94142e10380 not found: ID does not exist" containerID="1132d8b5d2ee66fa77fee0834a87abe99effceb61968ba705b67a94142e10380"
Jul 15 12:04:20.749078 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.749024    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1132d8b5d2ee66fa77fee0834a87abe99effceb61968ba705b67a94142e10380"} err="failed to get container status \"1132d8b5d2ee66fa77fee0834a87abe99effceb61968ba705b67a94142e10380\": rpc error: code = NotFound desc = could not find container \"1132d8b5d2ee66fa77fee0834a87abe99effceb61968ba705b67a94142e10380\": container with ID starting with 1132d8b5d2ee66fa77fee0834a87abe99effceb61968ba705b67a94142e10380 not found: ID does not exist"
Jul 15 12:04:20.758552 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.758528    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"]
Jul 15 12:04:20.763029 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:20.763009    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-cd94698b5-9kzv7"]
Jul 15 12:04:21.270710 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:21.268572    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c6cd081-4674-49f3-abf2-6f1efc5de7ab" path="/var/lib/kubelet/pods/7c6cd081-4674-49f3-abf2-6f1efc5de7ab/volumes"
Jul 15 12:04:24.023712 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:24.023670    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 15 12:04:34.024430 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:34.024373    2566 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 15 12:04:44.033780 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:44.033750    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:04:44.041487 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:44.041463    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:04:47.179504 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:04:47.179458    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"]
Jul 15 12:05:02.202815 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.202764    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerName="main" containerID="cri-o://075020b843f2fa2b3f4cb20906af8dfc9834ea1b1ac617ae258c34cbfb789bb7" gracePeriod=45
Jul 15 12:05:02.852256 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.852227    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:05:02.922053 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.922016    2566 generic.go:356] "Generic (PLEG): container finished" podID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerID="075020b843f2fa2b3f4cb20906af8dfc9834ea1b1ac617ae258c34cbfb789bb7" exitCode=0
Jul 15 12:05:02.922246 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.922065    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"
Jul 15 12:05:02.922246 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.922089    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" event={"ID":"a62613a0-2162-45c9-86fb-c5c8977842d8","Type":"ContainerDied","Data":"075020b843f2fa2b3f4cb20906af8dfc9834ea1b1ac617ae258c34cbfb789bb7"}
Jul 15 12:05:02.922246 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.922144    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq" event={"ID":"a62613a0-2162-45c9-86fb-c5c8977842d8","Type":"ContainerDied","Data":"35c5b40fc2c9a11391ff0e30368fe835a202d5c10279a8a1eb056aeeaa955fa5"}
Jul 15 12:05:02.922246 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.922165    2566 scope.go:117] "RemoveContainer" containerID="075020b843f2fa2b3f4cb20906af8dfc9834ea1b1ac617ae258c34cbfb789bb7"
Jul 15 12:05:02.932670 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.932646    2566 scope.go:117] "RemoveContainer" containerID="2c56e49f8f9cc57fac7dd967e781ae4a922f559f6bb6204611038e18f0eae65d"
Jul 15 12:05:02.941240 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.941218    2566 scope.go:117] "RemoveContainer" containerID="075020b843f2fa2b3f4cb20906af8dfc9834ea1b1ac617ae258c34cbfb789bb7"
Jul 15 12:05:02.941559 ip-10-0-133-26 kubenswrapper[2566]: E0715 12:05:02.941538    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"075020b843f2fa2b3f4cb20906af8dfc9834ea1b1ac617ae258c34cbfb789bb7\": container with ID starting with 075020b843f2fa2b3f4cb20906af8dfc9834ea1b1ac617ae258c34cbfb789bb7 not found: ID does not exist" containerID="075020b843f2fa2b3f4cb20906af8dfc9834ea1b1ac617ae258c34cbfb789bb7"
Jul 15 12:05:02.941641 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.941574    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"075020b843f2fa2b3f4cb20906af8dfc9834ea1b1ac617ae258c34cbfb789bb7"} err="failed to get container status \"075020b843f2fa2b3f4cb20906af8dfc9834ea1b1ac617ae258c34cbfb789bb7\": rpc error: code = NotFound desc = could not find container \"075020b843f2fa2b3f4cb20906af8dfc9834ea1b1ac617ae258c34cbfb789bb7\": container with ID starting with 075020b843f2fa2b3f4cb20906af8dfc9834ea1b1ac617ae258c34cbfb789bb7 not found: ID does not exist"
Jul 15 12:05:02.941641 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.941601    2566 scope.go:117] "RemoveContainer" containerID="2c56e49f8f9cc57fac7dd967e781ae4a922f559f6bb6204611038e18f0eae65d"
Jul 15 12:05:02.941869 ip-10-0-133-26 kubenswrapper[2566]: E0715 12:05:02.941852    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c56e49f8f9cc57fac7dd967e781ae4a922f559f6bb6204611038e18f0eae65d\": container with ID starting with 2c56e49f8f9cc57fac7dd967e781ae4a922f559f6bb6204611038e18f0eae65d not found: ID does not exist" containerID="2c56e49f8f9cc57fac7dd967e781ae4a922f559f6bb6204611038e18f0eae65d"
Jul 15 12:05:02.941909 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.941876    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c56e49f8f9cc57fac7dd967e781ae4a922f559f6bb6204611038e18f0eae65d"} err="failed to get container status \"2c56e49f8f9cc57fac7dd967e781ae4a922f559f6bb6204611038e18f0eae65d\": rpc error: code = NotFound desc = could not find container \"2c56e49f8f9cc57fac7dd967e781ae4a922f559f6bb6204611038e18f0eae65d\": container with ID starting with 2c56e49f8f9cc57fac7dd967e781ae4a922f559f6bb6204611038e18f0eae65d not found: ID does not exist"
Jul 15 12:05:02.944184 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.944165    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-kserve-provision-location\") pod \"a62613a0-2162-45c9-86fb-c5c8977842d8\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") "
Jul 15 12:05:02.944246 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.944219    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-dshm\") pod \"a62613a0-2162-45c9-86fb-c5c8977842d8\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") "
Jul 15 12:05:02.944307 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.944247    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-tmp-dir\") pod \"a62613a0-2162-45c9-86fb-c5c8977842d8\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") "
Jul 15 12:05:02.944307 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.944291    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-model-cache\") pod \"a62613a0-2162-45c9-86fb-c5c8977842d8\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") "
Jul 15 12:05:02.944412 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.944310    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v44wx\" (UniqueName: \"kubernetes.io/projected/a62613a0-2162-45c9-86fb-c5c8977842d8-kube-api-access-v44wx\") pod \"a62613a0-2162-45c9-86fb-c5c8977842d8\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") "
Jul 15 12:05:02.944412 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.944330    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62613a0-2162-45c9-86fb-c5c8977842d8-tls-certs\") pod \"a62613a0-2162-45c9-86fb-c5c8977842d8\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") "
Jul 15 12:05:02.944412 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.944352    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-home\") pod \"a62613a0-2162-45c9-86fb-c5c8977842d8\" (UID: \"a62613a0-2162-45c9-86fb-c5c8977842d8\") "
Jul 15 12:05:02.944683 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.944635    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "a62613a0-2162-45c9-86fb-c5c8977842d8" (UID: "a62613a0-2162-45c9-86fb-c5c8977842d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:05:02.945172 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.945112    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-home" (OuterVolumeSpecName: "home") pod "a62613a0-2162-45c9-86fb-c5c8977842d8" (UID: "a62613a0-2162-45c9-86fb-c5c8977842d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:05:02.946729 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.946705    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-dshm" (OuterVolumeSpecName: "dshm") pod "a62613a0-2162-45c9-86fb-c5c8977842d8" (UID: "a62613a0-2162-45c9-86fb-c5c8977842d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:05:02.946891 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.946872    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62613a0-2162-45c9-86fb-c5c8977842d8-kube-api-access-v44wx" (OuterVolumeSpecName: "kube-api-access-v44wx") pod "a62613a0-2162-45c9-86fb-c5c8977842d8" (UID: "a62613a0-2162-45c9-86fb-c5c8977842d8"). InnerVolumeSpecName "kube-api-access-v44wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 12:05:02.947613 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.947596    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62613a0-2162-45c9-86fb-c5c8977842d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a62613a0-2162-45c9-86fb-c5c8977842d8" (UID: "a62613a0-2162-45c9-86fb-c5c8977842d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 12:05:02.956718 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:02.956693    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a62613a0-2162-45c9-86fb-c5c8977842d8" (UID: "a62613a0-2162-45c9-86fb-c5c8977842d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:05:03.022601 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:03.022563    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a62613a0-2162-45c9-86fb-c5c8977842d8" (UID: "a62613a0-2162-45c9-86fb-c5c8977842d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:05:03.045290 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:03.045251    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-dshm\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 12:05:03.045290 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:03.045280    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-tmp-dir\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 12:05:03.045290 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:03.045289    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-model-cache\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 12:05:03.045561 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:03.045300    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v44wx\" (UniqueName: \"kubernetes.io/projected/a62613a0-2162-45c9-86fb-c5c8977842d8-kube-api-access-v44wx\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 12:05:03.045561 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:03.045309    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62613a0-2162-45c9-86fb-c5c8977842d8-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 12:05:03.045561 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:03.045316    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-home\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 12:05:03.045561 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:03.045324    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a62613a0-2162-45c9-86fb-c5c8977842d8-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 12:05:03.244421 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:03.244383    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"]
Jul 15 12:05:03.248294 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:03.248266    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-85ccc87949-2bshq"]
Jul 15 12:05:03.267715 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:03.267672    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" path="/var/lib/kubelet/pods/a62613a0-2162-45c9-86fb-c5c8977842d8/volumes"
Jul 15 12:05:03.514816 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:03.514776    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"]
Jul 15 12:05:08.283391 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:08.283359    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8" podUID="87acd944-f0a2-491c-8d52-fccb70099d0a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 12:05:18.283307 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.283268    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8" podUID="87acd944-f0a2-491c-8d52-fccb70099d0a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 15 12:05:18.535890 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.535789    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8" podUID="87acd944-f0a2-491c-8d52-fccb70099d0a" containerName="main" containerID="cri-o://ca0a03568dc396ca9a5fba3d6018625cd29ccb3b50ec787fe8bc32547debfb80" gracePeriod=45
Jul 15 12:05:18.789194 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.789139    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:05:18.854137 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.854106    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/87acd944-f0a2-491c-8d52-fccb70099d0a-tokenizer-uds\") pod \"87acd944-f0a2-491c-8d52-fccb70099d0a\" (UID: \"87acd944-f0a2-491c-8d52-fccb70099d0a\") "
Jul 15 12:05:18.854300 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.854191    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7cvx\" (UniqueName: \"kubernetes.io/projected/87acd944-f0a2-491c-8d52-fccb70099d0a-kube-api-access-k7cvx\") pod \"87acd944-f0a2-491c-8d52-fccb70099d0a\" (UID: \"87acd944-f0a2-491c-8d52-fccb70099d0a\") "
Jul 15 12:05:18.854300 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.854213    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87acd944-f0a2-491c-8d52-fccb70099d0a-tls-certs\") pod \"87acd944-f0a2-491c-8d52-fccb70099d0a\" (UID: \"87acd944-f0a2-491c-8d52-fccb70099d0a\") "
Jul 15 12:05:18.854300 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.854236    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87acd944-f0a2-491c-8d52-fccb70099d0a-kserve-provision-location\") pod \"87acd944-f0a2-491c-8d52-fccb70099d0a\" (UID: \"87acd944-f0a2-491c-8d52-fccb70099d0a\") "
Jul 15 12:05:18.854465 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.854401    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87acd944-f0a2-491c-8d52-fccb70099d0a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "87acd944-f0a2-491c-8d52-fccb70099d0a" (UID: "87acd944-f0a2-491c-8d52-fccb70099d0a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:05:18.855008 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.854984    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87acd944-f0a2-491c-8d52-fccb70099d0a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "87acd944-f0a2-491c-8d52-fccb70099d0a" (UID: "87acd944-f0a2-491c-8d52-fccb70099d0a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 15 12:05:18.856485 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.856463    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87acd944-f0a2-491c-8d52-fccb70099d0a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87acd944-f0a2-491c-8d52-fccb70099d0a" (UID: "87acd944-f0a2-491c-8d52-fccb70099d0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 15 12:05:18.856940 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.856521    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87acd944-f0a2-491c-8d52-fccb70099d0a-kube-api-access-k7cvx" (OuterVolumeSpecName: "kube-api-access-k7cvx") pod "87acd944-f0a2-491c-8d52-fccb70099d0a" (UID: "87acd944-f0a2-491c-8d52-fccb70099d0a"). InnerVolumeSpecName "kube-api-access-k7cvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 15 12:05:18.955391 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.955357    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7cvx\" (UniqueName: \"kubernetes.io/projected/87acd944-f0a2-491c-8d52-fccb70099d0a-kube-api-access-k7cvx\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 12:05:18.955391 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.955385    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87acd944-f0a2-491c-8d52-fccb70099d0a-tls-certs\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 12:05:18.955391 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.955394    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87acd944-f0a2-491c-8d52-fccb70099d0a-kserve-provision-location\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 12:05:18.955391 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.955402    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/87acd944-f0a2-491c-8d52-fccb70099d0a-tokenizer-uds\") on node \"ip-10-0-133-26.ec2.internal\" DevicePath \"\""
Jul 15 12:05:18.993164 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.993132    2566 generic.go:356] "Generic (PLEG): container finished" podID="87acd944-f0a2-491c-8d52-fccb70099d0a" containerID="ca0a03568dc396ca9a5fba3d6018625cd29ccb3b50ec787fe8bc32547debfb80" exitCode=0
Jul 15 12:05:18.993327 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.993187    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"
Jul 15 12:05:18.993327 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.993205    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8" event={"ID":"87acd944-f0a2-491c-8d52-fccb70099d0a","Type":"ContainerDied","Data":"ca0a03568dc396ca9a5fba3d6018625cd29ccb3b50ec787fe8bc32547debfb80"}
Jul 15 12:05:18.993327 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.993254    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8" event={"ID":"87acd944-f0a2-491c-8d52-fccb70099d0a","Type":"ContainerDied","Data":"a577f4685c8fa4c6db812437b8de517d1beaf442654b7210aae9a79c12cb43ea"}
Jul 15 12:05:18.993327 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:18.993281    2566 scope.go:117] "RemoveContainer" containerID="ca0a03568dc396ca9a5fba3d6018625cd29ccb3b50ec787fe8bc32547debfb80"
Jul 15 12:05:19.002226 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:19.002206    2566 scope.go:117] "RemoveContainer" containerID="92abe81900d7ab7254bd53e88f1c3d70793c12a94fc105156c04329b6873adcd"
Jul 15 12:05:19.009839 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:19.009820    2566 scope.go:117] "RemoveContainer" containerID="ca0a03568dc396ca9a5fba3d6018625cd29ccb3b50ec787fe8bc32547debfb80"
Jul 15 12:05:19.010097 ip-10-0-133-26 kubenswrapper[2566]: E0715 12:05:19.010079    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0a03568dc396ca9a5fba3d6018625cd29ccb3b50ec787fe8bc32547debfb80\": container with ID starting with ca0a03568dc396ca9a5fba3d6018625cd29ccb3b50ec787fe8bc32547debfb80 not found: ID does not exist" containerID="ca0a03568dc396ca9a5fba3d6018625cd29ccb3b50ec787fe8bc32547debfb80"
Jul 15 12:05:19.010139 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:19.010107    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0a03568dc396ca9a5fba3d6018625cd29ccb3b50ec787fe8bc32547debfb80"} err="failed to get container status \"ca0a03568dc396ca9a5fba3d6018625cd29ccb3b50ec787fe8bc32547debfb80\": rpc error: code = NotFound desc = could not find container \"ca0a03568dc396ca9a5fba3d6018625cd29ccb3b50ec787fe8bc32547debfb80\": container with ID starting with ca0a03568dc396ca9a5fba3d6018625cd29ccb3b50ec787fe8bc32547debfb80 not found: ID does not exist"
Jul 15 12:05:19.010139 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:19.010125    2566 scope.go:117] "RemoveContainer" containerID="92abe81900d7ab7254bd53e88f1c3d70793c12a94fc105156c04329b6873adcd"
Jul 15 12:05:19.010363 ip-10-0-133-26 kubenswrapper[2566]: E0715 12:05:19.010345    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92abe81900d7ab7254bd53e88f1c3d70793c12a94fc105156c04329b6873adcd\": container with ID starting with 92abe81900d7ab7254bd53e88f1c3d70793c12a94fc105156c04329b6873adcd not found: ID does not exist" containerID="92abe81900d7ab7254bd53e88f1c3d70793c12a94fc105156c04329b6873adcd"
Jul 15 12:05:19.010397 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:19.010369    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92abe81900d7ab7254bd53e88f1c3d70793c12a94fc105156c04329b6873adcd"} err="failed to get container status \"92abe81900d7ab7254bd53e88f1c3d70793c12a94fc105156c04329b6873adcd\": rpc error: code = NotFound desc = could not find container \"92abe81900d7ab7254bd53e88f1c3d70793c12a94fc105156c04329b6873adcd\": container with ID starting with 92abe81900d7ab7254bd53e88f1c3d70793c12a94fc105156c04329b6873adcd not found: ID does not exist"
Jul 15 12:05:19.018681 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:19.018660    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"]
Jul 15 12:05:19.023298 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:19.023277    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-7d847d9b5-hw4s8"]
Jul 15 12:05:19.267662 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:19.267631    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87acd944-f0a2-491c-8d52-fccb70099d0a" path="/var/lib/kubelet/pods/87acd944-f0a2-491c-8d52-fccb70099d0a/volumes"
Jul 15 12:05:38.603245 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:38.603210    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 12:05:38.606980 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:05:38.606951    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5tqtw_20239449-cde8-4cb1-9c0f-dad9abb7cd58/ovn-acl-logging/0.log"
Jul 15 12:06:27.527985 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:27.527889    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/1.log"
Jul 15 12:06:27.528713 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:27.528696    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/0.log"
Jul 15 12:06:29.005675 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:29.005639    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/1.log"
Jul 15 12:06:29.006062 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:29.005759    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/0.log"
Jul 15 12:06:30.494828 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:30.494792    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/1.log"
Jul 15 12:06:30.495220 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:30.494954    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/0.log"
Jul 15 12:06:31.915641 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:31.915607    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/1.log"
Jul 15 12:06:31.916140 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:31.915777    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/0.log"
Jul 15 12:06:33.374708 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:33.374679    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/1.log"
Jul 15 12:06:33.375168 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:33.374953    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/0.log"
Jul 15 12:06:34.834998 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:34.834963    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/1.log"
Jul 15 12:06:34.835473 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:34.835120    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/0.log"
Jul 15 12:06:36.370702 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:36.370631    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/1.log"
Jul 15 12:06:36.371124 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:36.370749    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/0.log"
Jul 15 12:06:37.897565 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:37.897533    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/1.log"
Jul 15 12:06:37.898046 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:37.897755    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/0.log"
Jul 15 12:06:39.509032 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:39.508990    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/1.log"
Jul 15 12:06:39.509485 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:39.509202    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/0.log"
Jul 15 12:06:40.968875 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:40.968809    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/1.log"
Jul 15 12:06:40.969355 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:40.968939    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/0.log"
Jul 15 12:06:42.378566 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:42.378530    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/1.log"
Jul 15 12:06:42.378999 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:42.378788    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/0.log"
Jul 15 12:06:43.805127 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:43.805101    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/1.log"
Jul 15 12:06:43.805527 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:43.805174    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/0.log"
Jul 15 12:06:45.231442 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:45.231409    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/1.log"
Jul 15 12:06:45.231442 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:45.231431    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/0.log"
Jul 15 12:06:46.664512 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:46.664470    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/1.log"
Jul 15 12:06:46.664994 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:46.664738    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/0.log"
Jul 15 12:06:48.123276 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:48.123243    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/1.log"
Jul 15 12:06:48.123736 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:48.123446    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/0.log"
Jul 15 12:06:49.624106 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:49.624077    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/1.log"
Jul 15 12:06:49.624553 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:49.624294    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-5f8bbcf7c7-7pkd7_53263949-66ba-4c6e-81dc-11865576a00c/istio-proxy/0.log"
Jul 15 12:06:52.044769 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:52.044741    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-fmhhm_63aad8c8-2687-4adf-9bab-9c36b0086419/istio-proxy/5.log"
Jul 15 12:06:52.069581 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:52.069554    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-fmhhm_63aad8c8-2687-4adf-9bab-9c36b0086419/istio-proxy/6.log"
Jul 15 12:06:54.028617 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:54.028585    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-fmhhm_63aad8c8-2687-4adf-9bab-9c36b0086419/istio-proxy/5.log"
Jul 15 12:06:54.052423 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:54.052397    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-fmhhm_63aad8c8-2687-4adf-9bab-9c36b0086419/istio-proxy/6.log"
Jul 15 12:06:55.034608 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:06:55.034575    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-twxjh_aab5fd40-59b4-4d7f-97d9-4ccf3a6f0c6d/authorino/0.log"
Jul 15 12:07:01.074487 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:01.074430    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5ffl8_dae7d2b5-9b3e-4ffe-85a3-087ad2313052/global-pull-secret-syncer/0.log"
Jul 15 12:07:01.275566 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:01.275520    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wcspb_cdfb31e6-b8ba-4726-afb8-6892dd4adcc7/konnectivity-agent/0.log"
Jul 15 12:07:01.332470 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:01.332394    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-26.ec2.internal_87afa8806d81c1e28dbf5cedd9e00a8e/haproxy/0.log"
Jul 15 12:07:05.756329 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:05.756295    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-twxjh_aab5fd40-59b4-4d7f-97d9-4ccf3a6f0c6d/authorino/0.log"
Jul 15 12:07:08.207192 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:08.207161    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jj947_cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4/node-exporter/0.log"
Jul 15 12:07:08.247155 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:08.247116    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jj947_cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4/kube-rbac-proxy/0.log"
Jul 15 12:07:08.270314 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:08.270293    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jj947_cc1d0e6f-d42d-401a-8f10-8f6eac3ba3c4/init-textfile/0.log"
Jul 15 12:07:10.921924 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.921892    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"]
Jul 15 12:07:10.922305 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.922184    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87acd944-f0a2-491c-8d52-fccb70099d0a" containerName="storage-initializer"
Jul 15 12:07:10.922305 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.922195    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="87acd944-f0a2-491c-8d52-fccb70099d0a" containerName="storage-initializer"
Jul 15 12:07:10.922305 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.922204    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6cd081-4674-49f3-abf2-6f1efc5de7ab" containerName="main"
Jul 15 12:07:10.922305 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.922209    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6cd081-4674-49f3-abf2-6f1efc5de7ab" containerName="main"
Jul 15 12:07:10.922305 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.922215    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerName="storage-initializer"
Jul 15 12:07:10.922305 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.922221    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerName="storage-initializer"
Jul 15 12:07:10.922305 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.922232    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerName="main"
Jul 15 12:07:10.922305 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.922238    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerName="main"
Jul 15 12:07:10.922305 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.922245    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6cd081-4674-49f3-abf2-6f1efc5de7ab" containerName="storage-initializer"
Jul 15 12:07:10.922305 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.922249    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6cd081-4674-49f3-abf2-6f1efc5de7ab" containerName="storage-initializer"
Jul 15 12:07:10.922305 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.922258    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87acd944-f0a2-491c-8d52-fccb70099d0a" containerName="main"
Jul 15 12:07:10.922305 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.922263    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="87acd944-f0a2-491c-8d52-fccb70099d0a" containerName="main"
Jul 15 12:07:10.922695 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.922314    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="87acd944-f0a2-491c-8d52-fccb70099d0a" containerName="main"
Jul 15 12:07:10.922695 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.922322    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6cd081-4674-49f3-abf2-6f1efc5de7ab" containerName="main"
Jul 15 12:07:10.922695 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.922330    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a62613a0-2162-45c9-86fb-c5c8977842d8" containerName="main"
Jul 15 12:07:10.924237 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.924221    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:10.926555 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.926534    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lt44n\"/\"openshift-service-ca.crt\""
Jul 15 12:07:10.927362 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.927347    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lt44n\"/\"kube-root-ca.crt\""
Jul 15 12:07:10.927449 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.927351    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lt44n\"/\"default-dockercfg-rg9nd\""
Jul 15 12:07:10.941545 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:10.941485    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"]
Jul 15 12:07:11.056650 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.056610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/112496da-0c84-4f30-b094-45e615d2cbf7-podres\") pod \"perf-node-gather-daemonset-ngk7n\" (UID: \"112496da-0c84-4f30-b094-45e615d2cbf7\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:11.056650 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.056651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/112496da-0c84-4f30-b094-45e615d2cbf7-proc\") pod \"perf-node-gather-daemonset-ngk7n\" (UID: \"112496da-0c84-4f30-b094-45e615d2cbf7\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:11.056865 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.056675    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/112496da-0c84-4f30-b094-45e615d2cbf7-sys\") pod \"perf-node-gather-daemonset-ngk7n\" (UID: \"112496da-0c84-4f30-b094-45e615d2cbf7\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:11.056865 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.056773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/112496da-0c84-4f30-b094-45e615d2cbf7-lib-modules\") pod \"perf-node-gather-daemonset-ngk7n\" (UID: \"112496da-0c84-4f30-b094-45e615d2cbf7\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:11.056865 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.056815    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbfqj\" (UniqueName: \"kubernetes.io/projected/112496da-0c84-4f30-b094-45e615d2cbf7-kube-api-access-cbfqj\") pod \"perf-node-gather-daemonset-ngk7n\" (UID: \"112496da-0c84-4f30-b094-45e615d2cbf7\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:11.158080 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.158040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/112496da-0c84-4f30-b094-45e615d2cbf7-lib-modules\") pod \"perf-node-gather-daemonset-ngk7n\" (UID: \"112496da-0c84-4f30-b094-45e615d2cbf7\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:11.158279 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.158087    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbfqj\" (UniqueName: \"kubernetes.io/projected/112496da-0c84-4f30-b094-45e615d2cbf7-kube-api-access-cbfqj\") pod \"perf-node-gather-daemonset-ngk7n\" (UID: \"112496da-0c84-4f30-b094-45e615d2cbf7\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:11.158279 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.158129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/112496da-0c84-4f30-b094-45e615d2cbf7-podres\") pod \"perf-node-gather-daemonset-ngk7n\" (UID: \"112496da-0c84-4f30-b094-45e615d2cbf7\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:11.158279 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.158146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/112496da-0c84-4f30-b094-45e615d2cbf7-proc\") pod \"perf-node-gather-daemonset-ngk7n\" (UID: \"112496da-0c84-4f30-b094-45e615d2cbf7\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:11.158279 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.158164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/112496da-0c84-4f30-b094-45e615d2cbf7-sys\") pod \"perf-node-gather-daemonset-ngk7n\" (UID: \"112496da-0c84-4f30-b094-45e615d2cbf7\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:11.158279 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.158229    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/112496da-0c84-4f30-b094-45e615d2cbf7-lib-modules\") pod \"perf-node-gather-daemonset-ngk7n\" (UID: \"112496da-0c84-4f30-b094-45e615d2cbf7\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:11.158279 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.158240    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/112496da-0c84-4f30-b094-45e615d2cbf7-sys\") pod \"perf-node-gather-daemonset-ngk7n\" (UID: \"112496da-0c84-4f30-b094-45e615d2cbf7\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:11.158279 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.158269    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/112496da-0c84-4f30-b094-45e615d2cbf7-proc\") pod \"perf-node-gather-daemonset-ngk7n\" (UID: \"112496da-0c84-4f30-b094-45e615d2cbf7\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:11.158563 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.158272    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/112496da-0c84-4f30-b094-45e615d2cbf7-podres\") pod \"perf-node-gather-daemonset-ngk7n\" (UID: \"112496da-0c84-4f30-b094-45e615d2cbf7\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:11.167735 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.167712    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbfqj\" (UniqueName: \"kubernetes.io/projected/112496da-0c84-4f30-b094-45e615d2cbf7-kube-api-access-cbfqj\") pod \"perf-node-gather-daemonset-ngk7n\" (UID: \"112496da-0c84-4f30-b094-45e615d2cbf7\") " pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:11.233898 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.233797    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:11.369402 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.369377    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"]
Jul 15 12:07:11.372127 ip-10-0-133-26 kubenswrapper[2566]: W0715 12:07:11.372096    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod112496da_0c84_4f30_b094_45e615d2cbf7.slice/crio-9f7d2614ea8c2cdf0f7e9a349973188b7a6d82f20ee845372731decd2c4d3852 WatchSource:0}: Error finding container 9f7d2614ea8c2cdf0f7e9a349973188b7a6d82f20ee845372731decd2c4d3852: Status 404 returned error can't find the container with id 9f7d2614ea8c2cdf0f7e9a349973188b7a6d82f20ee845372731decd2c4d3852
Jul 15 12:07:11.470898 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:11.470862    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n" event={"ID":"112496da-0c84-4f30-b094-45e615d2cbf7","Type":"ContainerStarted","Data":"9f7d2614ea8c2cdf0f7e9a349973188b7a6d82f20ee845372731decd2c4d3852"}
Jul 15 12:07:12.034908 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:12.034882    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-xftvp_5ca40d07-27fa-4bcb-8bc4-f01651105a6a/volume-data-source-validator/0.log"
Jul 15 12:07:12.476461 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:12.476352    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n" event={"ID":"112496da-0c84-4f30-b094-45e615d2cbf7","Type":"ContainerStarted","Data":"cd4063f4900a145487faa7d6486c38c561836b9648a23b8f6dfc26bb1c50c56b"}
Jul 15 12:07:12.476667 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:12.476516    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:12.495270 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:12.495221    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n" podStartSLOduration=2.495204705 podStartE2EDuration="2.495204705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:07:12.493807333 +0000 UTC m=+4901.723774864" watchObservedRunningTime="2026-07-15 12:07:12.495204705 +0000 UTC m=+4901.725172235"
Jul 15 12:07:13.119150 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:13.119116    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8f8hw_dfc3a194-a6c3-4cee-8f9c-225bc846d9b9/dns/0.log"
Jul 15 12:07:13.142199 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:13.142149    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8f8hw_dfc3a194-a6c3-4cee-8f9c-225bc846d9b9/kube-rbac-proxy/0.log"
Jul 15 12:07:13.287064 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:13.287032    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ww56m_6918f75f-21cc-47c8-9332-c1055bdbf5df/dns-node-resolver/0.log"
Jul 15 12:07:13.845357 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:13.845324    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-d7778dc9b-mh9g2_5ffa720a-c23e-44c7-84e8-95d7724446ba/registry/0.log"
Jul 15 12:07:13.998208 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:13.998172    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hxflc_c118fdb2-4d52-4883-99c6-cf8609b6d8dd/node-ca/0.log"
Jul 15 12:07:15.764544 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:15.764515    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-fmhhm_63aad8c8-2687-4adf-9bab-9c36b0086419/istio-proxy/5.log"
Jul 15 12:07:15.788442 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:15.788417    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-fmhhm_63aad8c8-2687-4adf-9bab-9c36b0086419/istio-proxy/6.log"
Jul 15 12:07:16.339682 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:16.339653    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b5jv4_599648bd-bc11-4e5a-9664-0c4892ce9edc/serve-healthcheck-canary/0.log"
Jul 15 12:07:16.901435 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:16.901409    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t8tvc_d30589a9-b904-4874-b34b-93d1ee5507d2/kube-rbac-proxy/0.log"
Jul 15 12:07:16.925634 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:16.925603    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t8tvc_d30589a9-b904-4874-b34b-93d1ee5507d2/exporter/0.log"
Jul 15 12:07:16.949483 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:16.949455    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t8tvc_d30589a9-b904-4874-b34b-93d1ee5507d2/extractor/0.log"
Jul 15 12:07:18.489970 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:18.489944    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lt44n/perf-node-gather-daemonset-ngk7n"
Jul 15 12:07:20.047522 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:20.047469    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6d9b94dcd4-b72nk_d7f58f48-9b6d-4577-afe4-e7ffe1f7eea5/manager/0.log"
Jul 15 12:07:20.813855 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:20.813825    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-749567458d-rwq52_3049f489-4d0e-45cd-8cda-3adae22e5975/manager/0.log"
Jul 15 12:07:20.851183 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:20.851151    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-798df69954-4m6w9_623186bb-08fc-42c1-92fa-4cc2989afccb/manager/0.log"
Jul 15 12:07:21.488142 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:21.488110    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9thnn_a476ce1b-19f5-42df-a54f-cd25aa1dbb66/s3-init/0.log"
Jul 15 12:07:21.509109 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:21.509080    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9thnn_a476ce1b-19f5-42df-a54f-cd25aa1dbb66/download-hf-model/0.log"
Jul 15 12:07:21.533011 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:21.532983    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-gzqrl_e2309ec5-89b9-40d9-9521-f67f88f4c4a1/seaweedfs/0.log"
Jul 15 12:07:23.053400 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:23.053365    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-fmhhm_63aad8c8-2687-4adf-9bab-9c36b0086419/istio-proxy/5.log"
Jul 15 12:07:23.078285 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:23.078253    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-fmhhm_63aad8c8-2687-4adf-9bab-9c36b0086419/istio-proxy/6.log"
Jul 15 12:07:24.935216 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:24.935187    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-fmhhm_63aad8c8-2687-4adf-9bab-9c36b0086419/istio-proxy/5.log"
Jul 15 12:07:24.959737 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:24.959710    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-fmhhm_63aad8c8-2687-4adf-9bab-9c36b0086419/istio-proxy/6.log"
Jul 15 12:07:26.727240 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:26.727210    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-drzdz_0b55cf39-c58b-4ef8-82b1-706516c52fc9/migrator/0.log"
Jul 15 12:07:26.750255 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:26.750233    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-drzdz_0b55cf39-c58b-4ef8-82b1-706516c52fc9/graceful-termination/0.log"
Jul 15 12:07:28.223251 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:28.223221    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4qs25_6321df80-acdb-482c-bb4e-d04645f9131f/kube-multus-additional-cni-plugins/0.log"
Jul 15 12:07:28.248321 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:28.248295    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4qs25_6321df80-acdb-482c-bb4e-d04645f9131f/egress-router-binary-copy/0.log"
Jul 15 12:07:28.270765 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:28.270740    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4qs25_6321df80-acdb-482c-bb4e-d04645f9131f/cni-plugins/0.log"
Jul 15 12:07:28.295880 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:28.295862    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4qs25_6321df80-acdb-482c-bb4e-d04645f9131f/bond-cni-plugin/0.log"
Jul 15 12:07:28.322065 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:28.322047    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4qs25_6321df80-acdb-482c-bb4e-d04645f9131f/routeoverride-cni/0.log"
Jul 15 12:07:28.344922 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:28.344898    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4qs25_6321df80-acdb-482c-bb4e-d04645f9131f/whereabouts-cni-bincopy/0.log"
Jul 15 12:07:28.367437 ip-10-0-133-26 kubenswrapper[2566]: I0715 12:07:28.367421    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4qs25_6321df80-acdb-482c-bb4e-d04645f9131f/whereabouts-cni/0.log"