Jul 31 14:46:18.713457 ip-10-0-133-32 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 31 14:46:18.713472 ip-10-0-133-32 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 31 14:46:18.713483 ip-10-0-133-32 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 31 14:46:18.713801 ip-10-0-133-32 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 31 14:46:28.735791 ip-10-0-133-32 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 31 14:46:28.735810 ip-10-0-133-32 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 36593933521e45d5a3a6272267ccc417 --
Jul 31 14:48:21.874649 ip-10-0-133-32 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 14:48:22.270750 ip-10-0-133-32 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:48:22.270750 ip-10-0-133-32 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 14:48:22.270750 ip-10-0-133-32 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:48:22.270750 ip-10-0-133-32 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 14:48:22.270750 ip-10-0-133-32 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:48:22.271463 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.270802    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 14:48:22.275123 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275106    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:22.275123 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275122    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275126    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275130    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275133    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275136    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275139    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275142    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275145    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275148    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275151    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275154    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275156    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275159    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275162    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275165    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275167    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275170    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275172    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275175    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275183    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:22.275186 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275186    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275190    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275195    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275198    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275201    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275204    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275207    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275209    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275212    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275215    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275217    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275221    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275225    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275228    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275231    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275234    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275236    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275241    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275245    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:22.275790 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275248    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275251    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275254    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275256    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275259    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275261    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275264    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275267    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275269    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275271    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275274    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275277    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275279    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275282    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275284    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275288    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275291    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275293    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275296    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275299    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:22.276296 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275303    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275305    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275308    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275311    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275314    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275316    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275319    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275321    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275324    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275326    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275330    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275333    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275336    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275338    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275341    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275344    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275356    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275358    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275361    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275364    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:22.276820 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275366    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275369    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275372    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275374    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275377    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275379    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275382    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275386    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275389    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275391    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275394    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275397    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275399    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275402    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275404    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275407    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275409    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275412    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.275414    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:22.277311 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276109    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276115    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276118    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276121    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276123    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276127    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276129    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276134    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276138    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276141    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276144    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276147    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276150    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276153    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276156    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276159    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276161    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276164    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276167    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276170    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:22.277794 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276175    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276178    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276181    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276183    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276187    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276191    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276193    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276196    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276199    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276202    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276204    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276207    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276210    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276212    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276215    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276217    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276220    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276222    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276226    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276228    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:22.278292 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276231    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276233    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276236    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276238    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276241    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276243    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276246    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276248    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276250    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276254    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276257    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276260    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276262    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276265    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276268    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276270    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276273    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276275    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276278    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276280    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:22.278847 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276283    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276285    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276288    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276291    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276293    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276296    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276298    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276301    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276305    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276308    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276310    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276313    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276316    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276319    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276321    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276324    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276327    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276329    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276332    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:22.279355 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276334    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276337    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276339    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276342    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276344    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276366    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276369    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276372    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276375    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276377    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276380    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276383    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276387    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276390    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276393    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276396    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276398    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276401    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276403    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276406    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:22.279828 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276815    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276819    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276823    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276826    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276829    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276832    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276836    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276840    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276843    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276846    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276849    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276852    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276855    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276858    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276861    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276864    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276866    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276869    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276872    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:22.280317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276875    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276878    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276880    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276883    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276885    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276888    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276891    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276895    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276898    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276901    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276904    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276906    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276909    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276911    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276914    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276919    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276922    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276924    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276928    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276930    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:22.280807 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276933    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276936    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276938    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276941    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276943    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276946    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276949    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276952    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276955    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276958    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276961    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276964    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276966    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276969    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276971    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276974    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276976    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276979    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276983    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:22.281338 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276986    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276989    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276991    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276993    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276996    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.276998    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277001    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277004    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277007    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277010    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277013    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277015    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277018    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277020    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277023    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277026    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277028    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277031    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277033    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277035    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:22.281826 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277038    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277041    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277044    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277047    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277050    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277052    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277055    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277057    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277060    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277062    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277065    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277068    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277070    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277073    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277075    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277078    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277080    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277083    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277085    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277088    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277091    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:22.282316 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277165    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277177    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277183    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277188    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277192    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277196    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277200    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277205    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277208    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277211    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277214    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277217    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277221    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277224    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277227    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277230    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277233    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277236    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277243    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277246    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277249    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277252    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277256    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277259    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 14:48:22.282868 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277262    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277265    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277269    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277272    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277275    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277278    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277281    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277285    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277289    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277292    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277295    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277298    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277301    2578 flags.go:64] FLAG: --enable-server="true"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277305    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277309    2578 flags.go:64] FLAG: --event-burst="100"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277312    2578 flags.go:64] FLAG: --event-qps="50"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277315    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277319    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277322    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277325    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277328    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277331    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277335    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277338    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277341    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 14:48:22.283485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277344    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277358    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277362    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277365    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277368    2578 flags.go:64] FLAG: --feature-gates=""
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277372    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277375    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277378    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277381    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277384    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277387    2578 flags.go:64] FLAG: --help="false"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277390    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277393    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277396    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277401    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277404    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277408    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277411    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277414    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277416    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277419    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277423    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277426    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277429    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 31 14:48:22.284121 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277432    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277435    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277441    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277444    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277447    2578 flags.go:64] FLAG: --lock-file=""
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277450    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277453    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277456    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277461    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277464    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277467    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277470    2578 flags.go:64] FLAG: --logging-format="text"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277473    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277476    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277479    2578 flags.go:64] FLAG: --manifest-url=""
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277482    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277487    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277490    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277494    2578 flags.go:64] FLAG: --max-pods="110"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277497    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277500    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277503    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277508    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277511    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277514    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 14:48:22.284724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277517    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277524    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277527    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277530    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277533    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277537    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277543    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277546    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277549    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277552    2578 flags.go:64] FLAG: --port="10250"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277556    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277559    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09ce30fef9e4e430b"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277562    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277565    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277568    2578 flags.go:64] FLAG: --register-node="true"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277571    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277574    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277578    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277581    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277584    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277587    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277590    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277593    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277596    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277599    2578 flags.go:64] FLAG: --runonce="false"
Jul 31 14:48:22.285366 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277602    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277605    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277608    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277610    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277615    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277618    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277621    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277624    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277627    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277630    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277633    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277636    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277639    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277642    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277645    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277651    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277654    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277658    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277662    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277665    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277668    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277671    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277674    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277677    2578 flags.go:64] FLAG: --v="2"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277681    2578 flags.go:64] FLAG: --version="false"
Jul 31 14:48:22.285981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277685    2578 flags.go:64] FLAG: --vmodule=""
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277689    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.277692    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277812    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277817    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277820    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277824    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277827    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277830    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277833    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277836    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277840    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277842    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277845    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277847    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277850    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277852    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277855    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277857    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277860    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:22.286625 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277863    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277866    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277868    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277871    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277875    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277878    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277880    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277883    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277886    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277889    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277891    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277894    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277896    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277899    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277902    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277904    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277908    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277911    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277914    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:22.287118 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277917    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277920    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277922    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277924    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277927    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277930    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277933    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277935    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277938    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277940    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277943    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277945    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277948    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277950    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277953    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277956    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277959    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277962    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277965    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277968    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:22.287600 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277970    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277973    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277975    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277978    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277981    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277983    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277986    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277988    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277991    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277993    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277996    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.277998    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278001    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278004    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278006    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278009    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278012    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278015    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278019    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:22.288124 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278021    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278024    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278026    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278029    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278032    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278034    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278037    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278039    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278042    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278045    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278049    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278052    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278054    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278057    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278060    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278062    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278064    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278067    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278070    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278072    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:22.288651 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278075    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:22.289174 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278078    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:22.289174 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278080    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:22.289174 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.278083    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:22.289174 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.278824    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:48:22.289174 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.285587    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 14:48:22.289174 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.285600    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 14:48:22.289174 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.286344    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 14:48:22.289174 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.286363    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 14:48:22.289174 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286421    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:22.289174 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286426    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:22.289174 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286429    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:22.289174 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286432    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:22.289174 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286434    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:22.289174 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286437    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286440    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286442    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286445    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286448    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286451    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286453    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286457    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286460    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286463    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286466    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286470    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286474    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286477    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286480    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286483    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286486    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286489    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286491    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:22.289652 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286494    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286497    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286499    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286502    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286504    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286507    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286511    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286513    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286516    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286518    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286522    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286526    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286529    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286531    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286536    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286539    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286542    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286545    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286547    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:22.290120 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286550    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286552    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286555    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286558    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286561    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286563    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286566    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286568    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286571    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286574    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286576    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286578    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286581    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286583    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286586    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286588    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286591    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286593    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286596    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286598    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:22.290603 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286601    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286604    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286606    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286609    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286611    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286614    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286616    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286618    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286621    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286623    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286626    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286629    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286631    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286634    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286636    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286639    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286642    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286645    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286648    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286650    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:22.291089 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286653    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:22.291700 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286655    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:22.291700 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286657    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:22.291700 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286660    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:22.291700 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286663    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:22.291700 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286665    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:22.291700 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286668    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:22.291700 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286670    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:22.291700 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286673    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:22.291700 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286675    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:22.291700 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286678    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:22.291700 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286680    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:22.291700 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286683    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:22.291700 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286686    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:22.291700 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286688    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:22.291700 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286690    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:22.292407 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.286695    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:48:22.292407 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286814    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:22.292407 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286818    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:22.292407 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286821    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:22.292407 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286824    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:22.292407 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286826    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:22.292407 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286829    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:22.292407 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286831    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:22.292407 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286834    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:22.292407 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286836    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:22.292407 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286839    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:22.292407 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286841    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:22.292407 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286845    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:22.292407 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286848    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:22.292407 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286850    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286853    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286855    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286858    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286860    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286863    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286865    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286868    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286870    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286873    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286875    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286878    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286880    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286883    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286885    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286888    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286890    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286893    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286896    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286898    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:22.293012 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286901    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286903    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286905    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286908    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286910    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286913    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286915    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286918    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286920    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286923    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286926    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286929    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286932    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286934    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286937    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286939    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286941    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286944    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286946    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286949    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:22.293549 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286951    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286954    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286956    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286960    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286964    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286967    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286969    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286972    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286974    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286977    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286979    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286982    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286984    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286987    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286990    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286994    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286997    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.286999    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287001    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:22.294033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287004    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287007    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287009    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287014    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287016    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287019    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287022    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287025    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287027    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287030    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287032    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287035    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287037    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287040    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287042    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287045    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287047    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287050    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287052    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287055    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:22.294514 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287057    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:22.295000 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287060    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:22.295000 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287062    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:22.295000 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287065    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:22.295000 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287067    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:22.295000 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287070    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:22.295000 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:22.287074    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:22.295000 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.287079    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:48:22.295000 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.287194    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 14:48:22.295000 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.289267    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 14:48:22.295000 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.290607    2578 server.go:1019] "Starting client certificate rotation"
Jul 31 14:48:22.295000 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.290709    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 14:48:22.295000 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.290751    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 14:48:22.295299 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.295102    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 14:48:22.299206 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.299186    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 14:48:22.311178 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.311160    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 31 14:48:22.313246 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.313227    2578 log.go:25] "Validated CRI v1 image API"
Jul 31 14:48:22.314050 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.314024    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 14:48:22.316472 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.316454    2578 fs.go:135] Filesystem UUIDs: map[13b1dc0c-c584-4971-98c6-f562cc8f9523:/dev/nvme0n1p3 4ba220cf-d0b8-4acf-86b0-7e6ec51fac60:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 31 14:48:22.316545 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.316471    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 14:48:22.322265 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.322155    2578 manager.go:216] Machine: {Timestamp:2026-07-31 14:48:22.320173623 +0000 UTC m=+0.329385482 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100053 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a2d98e4af711fa335a060d77a05c5 SystemUUID:ec2a2d98-e4af-711f-a335-a060d77a05c5 BootID:36593933-521e-45d5-a3a6-272267ccc417 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bf:a2:7e:04:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bf:a2:7e:04:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:83:19:a2:45:89 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 14:48:22.322265 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.322258    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 14:48:22.322416 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.322398    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 14:48:22.322784 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.322769    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 14:48:22.325662 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.325634    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 14:48:22.325803 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.325668    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-32.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 14:48:22.326377 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.326366    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 14:48:22.326409 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.326379    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 14:48:22.326409 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.326386    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 14:48:22.327081 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.327071    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 14:48:22.328547 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.328536    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 14:48:22.332656 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.332644    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 14:48:22.332791 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.332780    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 14:48:22.332822 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.332812    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 14:48:22.332849 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.332827    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 14:48:22.332849 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.332846    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 14:48:22.332899 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.332859    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 31 14:48:22.332899 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.332871    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 14:48:22.333330 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.333317    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 14:48:22.333388 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.333338    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 14:48:22.336941 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.336917    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 14:48:22.338703 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.338690    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 14:48:22.338749 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.338720    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 14:48:22.339532 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.339521    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 14:48:22.339574 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.339539    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 14:48:22.339574 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.339549    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 14:48:22.339574 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.339560    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 14:48:22.339574 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.339571    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 14:48:22.339679 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.339578    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 14:48:22.339679 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.339585    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 14:48:22.339679 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.339590    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 14:48:22.339679 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.339597    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 14:48:22.339679 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.339603    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 14:48:22.339679 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.339612    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 14:48:22.339679 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.339617    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 14:48:22.341149 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.341136    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 14:48:22.341149 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.341151    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 14:48:22.344404 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.344389    2578 server.go:1268] "Started kubelet"
Jul 31 14:48:22.344532 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.344491    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 14:48:22.344615 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.344484    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 14:48:22.344701 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.344673    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 14:48:22.345384 ip-10-0-133-32 systemd[1]: Started Kubernetes Kubelet.
Jul 31 14:48:22.345931 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.345914    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 14:48:22.346026 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.345916    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 14:48:22.348194 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.348165    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-32.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 14:48:22.348194 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.348159    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 14:48:22.348396 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.348377    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-32.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 14:48:22.348738 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.348724    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7l7sh"
Jul 31 14:48:22.351316 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.351298    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 14:48:22.351402 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.351325    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 14:48:22.351402 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.351324    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 14:48:22.351577 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.351557    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Jul 31 14:48:22.351868 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.350845    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-32.ec2.internal.18c767401d63de9c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-32.ec2.internal,UID:ip-10-0-133-32.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-32.ec2.internal,},FirstTimestamp:2026-07-31 14:48:22.344343196 +0000 UTC m=+0.353555057,LastTimestamp:2026-07-31 14:48:22.344343196 +0000 UTC m=+0.353555057,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-32.ec2.internal,}"
Jul 31 14:48:22.352009 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.351849    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 14:48:22.352009 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.352011    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 14:48:22.352129 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.351851    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 14:48:22.352160 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.352147    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 14:48:22.352160 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.352154    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 14:48:22.353063 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.353042    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-32.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 31 14:48:22.353063 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.353051    2578 factory.go:153] Registering CRI-O factory
Jul 31 14:48:22.353154 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.353074    2578 factory.go:223] Registration of the crio container factory successfully
Jul 31 14:48:22.353154 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.353120    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 14:48:22.353154 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.353128    2578 factory.go:55] Registering systemd factory
Jul 31 14:48:22.353154 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.353133    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 31 14:48:22.353154 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.353150    2578 factory.go:103] Registering Raw factory
Jul 31 14:48:22.353411 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.353147    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 31 14:48:22.353411 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.353164    2578 manager.go:1196] Started watching for new ooms in manager
Jul 31 14:48:22.353586 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.353568    2578 manager.go:349] Starting recovery of all containers
Jul 31 14:48:22.360328 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.360305    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7l7sh"
Jul 31 14:48:22.364569 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.364547    2578 manager.go:354] Recovery completed
Jul 31 14:48:22.368753 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.368740    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:22.370456 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.370438    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:22.370533 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.370469    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:22.370533 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.370479    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:22.371007 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.370995    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 14:48:22.371007 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.371006    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 14:48:22.371102 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.371011    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:22.371102 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.371021    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 14:48:22.371757 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.371741    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:22.371811 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.371769    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:22.371811 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.371781    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:22.372887 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.372874    2578 policy_none.go:49] "None policy: Start"
Jul 31 14:48:22.372937 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.372890    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 14:48:22.372937 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.372900    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 14:48:22.373003 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.372863    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-32.ec2.internal.18c767401ef24ea4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-32.ec2.internal,UID:ip-10-0-133-32.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-32.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-32.ec2.internal,},FirstTimestamp:2026-07-31 14:48:22.370455204 +0000 UTC m=+0.379667063,LastTimestamp:2026-07-31 14:48:22.370455204 +0000 UTC m=+0.379667063,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-32.ec2.internal,}"
Jul 31 14:48:22.374849 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.374837    2578 policy_none.go:47] "Start"
Jul 31 14:48:22.422208 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.421997    2578 manager.go:337] "Starting Device Plugin manager"
Jul 31 14:48:22.422314 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.422234    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 14:48:22.422314 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.422248    2578 server.go:87] "Starting device plugin registration server"
Jul 31 14:48:22.422569 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.422558    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 14:48:22.422601 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.422572    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 14:48:22.422690 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.422670    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 14:48:22.422774 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.422758    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 14:48:22.422774 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.422771    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 14:48:22.423339 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.423313    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 14:48:22.423425 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.423385    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-32.ec2.internal\" not found"
Jul 31 14:48:22.482655 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.482623    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 14:48:22.484761 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.483855    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 14:48:22.484761 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.483892    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 14:48:22.484761 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.483916    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 14:48:22.484761 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.483953    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 14:48:22.486102 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.486079    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:22.523225 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.523118    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:22.524069 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.524054    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:22.524150 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.524088    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:22.524150 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.524105    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:22.524150 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.524135    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.530644 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.530627    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.530715 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.530650    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-32.ec2.internal\": node \"ip-10-0-133-32.ec2.internal\" not found"
Jul 31 14:48:22.550623 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.550604    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Jul 31 14:48:22.584859 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.584820    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"]
Jul 31 14:48:22.584927 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.584905    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:22.585844 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.585829    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:22.585910 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.585857    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:22.585910 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.585871    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:22.588256 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.588242    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:22.588393 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.588378    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.588447 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.588416    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:22.589023 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.589004    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:22.589023 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.589018    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:22.589124 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.589035    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:22.589124 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.589046    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:22.589124 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.589051    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:22.589124 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.589069    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:22.593453 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.593436    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.593531 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.593463    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:22.594232 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.594216    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:22.594323 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.594245    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:22.594323 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.594259    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:22.614276 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.614255    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-32.ec2.internal\" not found" node="ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.619370 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.619350    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-32.ec2.internal\" not found" node="ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.652016 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.651989    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Jul 31 14:48:22.752094 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.752065    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Jul 31 14:48:22.753304 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.753285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7cb07b957eab05242e711549788246d2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"7cb07b957eab05242e711549788246d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.753359 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.753325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cb07b957eab05242e711549788246d2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"7cb07b957eab05242e711549788246d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.753359 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.753345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17290c901fea2fac12a7cbe3399d875c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-32.ec2.internal\" (UID: \"17290c901fea2fac12a7cbe3399d875c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.852247 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.852171    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Jul 31 14:48:22.854560 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.854540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7cb07b957eab05242e711549788246d2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"7cb07b957eab05242e711549788246d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.854616 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.854571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cb07b957eab05242e711549788246d2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"7cb07b957eab05242e711549788246d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.854616 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.854595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17290c901fea2fac12a7cbe3399d875c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-32.ec2.internal\" (UID: \"17290c901fea2fac12a7cbe3399d875c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.854682 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.854636    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7cb07b957eab05242e711549788246d2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"7cb07b957eab05242e711549788246d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.854682 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.854645    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17290c901fea2fac12a7cbe3399d875c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-32.ec2.internal\" (UID: \"17290c901fea2fac12a7cbe3399d875c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.854682 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.854645    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cb07b957eab05242e711549788246d2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"7cb07b957eab05242e711549788246d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.920765 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.920717    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.921911 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:22.921894    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Jul 31 14:48:22.952537 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:22.952505    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Jul 31 14:48:23.056161 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:23.056117    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Jul 31 14:48:23.156629 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:23.156550    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Jul 31 14:48:23.173448 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:23.171997    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:23.257489 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:23.257453    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Jul 31 14:48:23.291160 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:23.291122    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 14:48:23.291677 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:23.291325    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 14:48:23.291677 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:23.291326    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 14:48:23.357562 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:23.357521    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Jul 31 14:48:23.366494 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:23.366461    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 14:43:23 +0000 UTC" deadline="2028-03-14 07:29:18.364509054 +0000 UTC"
Jul 31 14:48:23.366494 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:23.366489    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14200h40m54.998023903s"
Jul 31 14:48:23.458713 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:23.458634    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Jul 31 14:48:23.477276 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:23.477251    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:23.559438 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:23.559405    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Jul 31 14:48:23.616982 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:23.616936    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17290c901fea2fac12a7cbe3399d875c.slice/crio-45b00130a2c144fd29027a500c833b23806a8afadf050048074bfc2af36462ca WatchSource:0}: Error finding container 45b00130a2c144fd29027a500c833b23806a8afadf050048074bfc2af36462ca: Status 404 returned error can't find the container with id 45b00130a2c144fd29027a500c833b23806a8afadf050048074bfc2af36462ca
Jul 31 14:48:23.621603 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:23.621579    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:48:23.659546 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:23.659508    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Jul 31 14:48:23.760577 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:23.760501    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Jul 31 14:48:23.764559 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:23.764542    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:23.851700 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:23.851662    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Jul 31 14:48:23.867694 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:23.867670    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 14:48:23.868774 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:23.868758    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Jul 31 14:48:23.876303 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:23.876283    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 14:48:23.917168 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:23.917139    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb07b957eab05242e711549788246d2.slice/crio-52baca642e76b303b1d185148e1b1d4099ea8bda94fed1d1e40fe010ed9220a8 WatchSource:0}: Error finding container 52baca642e76b303b1d185148e1b1d4099ea8bda94fed1d1e40fe010ed9220a8: Status 404 returned error can't find the container with id 52baca642e76b303b1d185148e1b1d4099ea8bda94fed1d1e40fe010ed9220a8
Jul 31 14:48:24.334284 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.334240    2578 apiserver.go:52] "Watching apiserver"
Jul 31 14:48:24.344840 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.344813    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 14:48:24.345338 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.345319    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-f74pb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal","openshift-multus/multus-additional-cni-plugins-4wc8q","openshift-multus/multus-sxt8b","kube-system/global-pull-secret-syncer-kg6bm","kube-system/konnectivity-agent-f4ptw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk","openshift-dns/node-resolver-x2h67","openshift-multus/network-metrics-daemon-97p2g","openshift-network-diagnostics/network-check-target-xs6tw","openshift-network-operator/iptables-alerter-fbgpj","openshift-ovn-kubernetes/ovnkube-node-lhnwk","kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal","openshift-cluster-node-tuning-operator/tuned-x4rzq"]
Jul 31 14:48:24.348082 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.348060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.350343 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.350316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fbgpj"
Jul 31 14:48:24.351172 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.351145    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 14:48:24.351172 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.351167    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 14:48:24.351327 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.351257    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-29w9c\""
Jul 31 14:48:24.351635 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.351502    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 14:48:24.351741 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.351646    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 14:48:24.352812 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.352791    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 14:48:24.353163 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.353144    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.353408 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.353388    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kqhlp\""
Jul 31 14:48:24.353739 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.353718    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:48:24.353832 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.353728    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 14:48:24.355298 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.355079    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.356048 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.356027    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 14:48:24.356315 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.356298    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 14:48:24.356557 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.356540    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cqlt8\""
Jul 31 14:48:24.356986 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.356779    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 14:48:24.356986 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.356806    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 14:48:24.356986 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.356815    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 14:48:24.359615 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.358247    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6kr7w\""
Jul 31 14:48:24.359615 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.358758    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 14:48:24.359771 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.359752    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:24.359871 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:24.359847    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:24.362831 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.362814    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f4ptw"
Jul 31 14:48:24.364113 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plmrw\" (UniqueName: \"kubernetes.io/projected/8f19f238-718d-44d6-86af-799a206bc26e-kube-api-access-plmrw\") pod \"iptables-alerter-fbgpj\" (UID: \"8f19f238-718d-44d6-86af-799a206bc26e\") " pod="openshift-network-operator/iptables-alerter-fbgpj"
Jul 31 14:48:24.364200 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d391192c-cdbd-46ce-9098-444df02b6f1b-cnibin\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.364200 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d391192c-cdbd-46ce-9098-444df02b6f1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.364200 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d391192c-cdbd-46ce-9098-444df02b6f1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.364352 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.364352 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-sys-fs\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.364352 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bmdm\" (UniqueName: \"kubernetes.io/projected/fce1aee2-b141-4232-8fc0-d34fb333f976-kube-api-access-8bmdm\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.364352 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d391192c-cdbd-46ce-9098-444df02b6f1b-system-cni-dir\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.364590 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d391192c-cdbd-46ce-9098-444df02b6f1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.364590 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-registration-dir\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.364590 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f19f238-718d-44d6-86af-799a206bc26e-iptables-alerter-script\") pod \"iptables-alerter-fbgpj\" (UID: \"8f19f238-718d-44d6-86af-799a206bc26e\") " pod="openshift-network-operator/iptables-alerter-fbgpj"
Jul 31 14:48:24.364590 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d391192c-cdbd-46ce-9098-444df02b6f1b-os-release\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.364590 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364528    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-socket-dir\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.364590 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364551    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-device-dir\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.364590 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-etc-selinux\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.364853 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364599    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f19f238-718d-44d6-86af-799a206bc26e-host-slash\") pod \"iptables-alerter-fbgpj\" (UID: \"8f19f238-718d-44d6-86af-799a206bc26e\") " pod="openshift-network-operator/iptables-alerter-fbgpj"
Jul 31 14:48:24.364853 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d391192c-cdbd-46ce-9098-444df02b6f1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.364853 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.364647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q47j9\" (UniqueName: \"kubernetes.io/projected/d391192c-cdbd-46ce-9098-444df02b6f1b-kube-api-access-q47j9\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.365697 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.365680    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:24.365779 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:24.365762    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:24.366026 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.366004    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 14:48:24.366493 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.366472    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rrpbl\""
Jul 31 14:48:24.367248 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.367211    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 14:48:24.367358 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.367333    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 14:48:24.368239 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.368220    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.370911 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.370756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x2h67"
Jul 31 14:48:24.371394 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.371375    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 14:48:24.371864 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.371846    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:48:24.371960 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.371904    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-49p62\""
Jul 31 14:48:24.373270 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.373250    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:24.373357 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:24.373329    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:24.373949 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.373929    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tk9qt\""
Jul 31 14:48:24.373949 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.373944    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 14:48:24.374090 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.373929    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 14:48:24.375715 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.375694    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f74pb"
Jul 31 14:48:24.378256 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.378236    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 14:48:24.378385 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.378365    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.378927 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.378908    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 14:48:24.378927 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.378923    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 14:48:24.379051 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.379036    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qjsqz\""
Jul 31 14:48:24.381343 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.381319    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 14:48:24.381343 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.381319    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 14:48:24.381607 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.381589    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 14:48:24.381684 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.381630    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xg67z\""
Jul 31 14:48:24.381684 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.381599    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 14:48:24.381898 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.381878    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 14:48:24.382149 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.382131    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 14:48:24.390546 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.390522    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2sc7k"
Jul 31 14:48:24.400540 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.400522    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2sc7k"
Jul 31 14:48:24.454264 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.454229    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 14:48:24.465661 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.465626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx4hj\" (UniqueName: \"kubernetes.io/projected/be585c19-1ea7-4e53-8142-da3d9b2bc15a-kube-api-access-cx4hj\") pod \"node-resolver-x2h67\" (UID: \"be585c19-1ea7-4e53-8142-da3d9b2bc15a\") " pod="openshift-dns/node-resolver-x2h67"
Jul 31 14:48:24.465835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.465675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp78b\" (UniqueName: \"kubernetes.io/projected/82144e0f-aab6-4481-ab7f-86a85e384b20-kube-api-access-vp78b\") pod \"node-ca-f74pb\" (UID: \"82144e0f-aab6-4481-ab7f-86a85e384b20\") " pod="openshift-image-registry/node-ca-f74pb"
Jul 31 14:48:24.465835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.465712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d391192c-cdbd-46ce-9098-444df02b6f1b-cnibin\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.465835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.465738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.465835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.465765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bmdm\" (UniqueName: \"kubernetes.io/projected/fce1aee2-b141-4232-8fc0-d34fb333f976-kube-api-access-8bmdm\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.465835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.465799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-run-openvswitch\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.465835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.465817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.465835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.465822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d391192c-cdbd-46ce-9098-444df02b6f1b-cnibin\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.466203 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.465835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-etc-kubernetes\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.466203 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.465892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/78ef4e44-0c0a-4875-9328-e83e1670ca8e-agent-certs\") pod \"konnectivity-agent-f4ptw\" (UID: \"78ef4e44-0c0a-4875-9328-e83e1670ca8e\") " pod="kube-system/konnectivity-agent-f4ptw"
Jul 31 14:48:24.466203 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.465920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be585c19-1ea7-4e53-8142-da3d9b2bc15a-hosts-file\") pod \"node-resolver-x2h67\" (UID: \"be585c19-1ea7-4e53-8142-da3d9b2bc15a\") " pod="openshift-dns/node-resolver-x2h67"
Jul 31 14:48:24.466203 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.465945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-kubelet\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.466203 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.465969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-run-netns\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.466203 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.465993    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-sysctl-conf\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.466203 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.465995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.466203 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be585c19-1ea7-4e53-8142-da3d9b2bc15a-tmp-dir\") pod \"node-resolver-x2h67\" (UID: \"be585c19-1ea7-4e53-8142-da3d9b2bc15a\") " pod="openshift-dns/node-resolver-x2h67"
Jul 31 14:48:24.466203 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f19f238-718d-44d6-86af-799a206bc26e-iptables-alerter-script\") pod \"iptables-alerter-fbgpj\" (UID: \"8f19f238-718d-44d6-86af-799a206bc26e\") " pod="openshift-network-operator/iptables-alerter-fbgpj"
Jul 31 14:48:24.466203 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-socket-dir\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.466203 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-system-cni-dir\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.466203 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-os-release\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.466203 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-run-netns\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.466203 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-run-multus-certs\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.466881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs\") pod \"network-metrics-daemon-97p2g\" (UID: \"352aa5e1-673c-4ede-a05a-80da4d81d2f9\") " pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:24.466881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-systemd-units\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.466881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466268    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c22c54bd-d71b-4169-8fc7-1810529a2e67-ovnkube-script-lib\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.466881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-socket-dir\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.466881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plmrw\" (UniqueName: \"kubernetes.io/projected/8f19f238-718d-44d6-86af-799a206bc26e-kube-api-access-plmrw\") pod \"iptables-alerter-fbgpj\" (UID: \"8f19f238-718d-44d6-86af-799a206bc26e\") " pod="openshift-network-operator/iptables-alerter-fbgpj"
Jul 31 14:48:24.466881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-sys-fs\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.466881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466402    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-node-log\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.466881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-log-socket\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.466881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466456    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-cni-netd\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.466881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-run-k8s-cni-cncf-io\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.466881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-var-lib-cni-bin\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.466881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d63bb96-09d0-4751-b541-9e8095b24c58-tmp\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.466881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466503    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-sys-fs\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.466881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d391192c-cdbd-46ce-9098-444df02b6f1b-system-cni-dir\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.466881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466574    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d391192c-cdbd-46ce-9098-444df02b6f1b-system-cni-dir\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.466881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-multus-conf-dir\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.466881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c3961a7e-b75a-4837-944f-330a900f1b75-multus-daemon-config\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.467835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-kubelet-config\") pod \"global-pull-secret-syncer-kg6bm\" (UID: \"7fae56cb-8b98-4d8b-a508-62dac7aa73d1\") " pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:24.467835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/82144e0f-aab6-4481-ab7f-86a85e384b20-serviceca\") pod \"node-ca-f74pb\" (UID: \"82144e0f-aab6-4481-ab7f-86a85e384b20\") " pod="openshift-image-registry/node-ca-f74pb"
Jul 31 14:48:24.467835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466687    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-run-ovn\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.467835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg282\" (UniqueName: \"kubernetes.io/projected/c22c54bd-d71b-4169-8fc7-1810529a2e67-kube-api-access-mg282\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.467835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-var-lib-kubelet\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.467835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxfdz\" (UniqueName: \"kubernetes.io/projected/c3961a7e-b75a-4837-944f-330a900f1b75-kube-api-access-wxfdz\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.467835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-systemd\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.467835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-etc-openvswitch\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.467835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466909    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-run-ovn-kubernetes\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.467835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c22c54bd-d71b-4169-8fc7-1810529a2e67-ovnkube-config\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.467835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.466962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-dbus\") pod \"global-pull-secret-syncer-kg6bm\" (UID: \"7fae56cb-8b98-4d8b-a508-62dac7aa73d1\") " pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:24.467835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d391192c-cdbd-46ce-9098-444df02b6f1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.467835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-lib-modules\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.467835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-host\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.467835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467083    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret\") pod \"global-pull-secret-syncer-kg6bm\" (UID: \"7fae56cb-8b98-4d8b-a508-62dac7aa73d1\") " pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:24.467835 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d391192c-cdbd-46ce-9098-444df02b6f1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.468576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-registration-dir\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.468576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d391192c-cdbd-46ce-9098-444df02b6f1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.468576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-registration-dir\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.468576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-multus-socket-dir-parent\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.468576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82144e0f-aab6-4481-ab7f-86a85e384b20-host\") pod \"node-ca-f74pb\" (UID: \"82144e0f-aab6-4481-ab7f-86a85e384b20\") " pod="openshift-image-registry/node-ca-f74pb"
Jul 31 14:48:24.468576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d391192c-cdbd-46ce-9098-444df02b6f1b-os-release\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.468576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-device-dir\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.468576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-etc-selinux\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.468576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-run-systemd\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.468576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d391192c-cdbd-46ce-9098-444df02b6f1b-os-release\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.468576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-var-lib-cni-multus\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.468576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-device-dir\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.468576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-sysconfig\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.468576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f19f238-718d-44d6-86af-799a206bc26e-host-slash\") pod \"iptables-alerter-fbgpj\" (UID: \"8f19f238-718d-44d6-86af-799a206bc26e\") " pod="openshift-network-operator/iptables-alerter-fbgpj"
Jul 31 14:48:24.468576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q47j9\" (UniqueName: \"kubernetes.io/projected/d391192c-cdbd-46ce-9098-444df02b6f1b-kube-api-access-q47j9\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.468576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fce1aee2-b141-4232-8fc0-d34fb333f976-etc-selinux\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.468576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467557    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f19f238-718d-44d6-86af-799a206bc26e-iptables-alerter-script\") pod \"iptables-alerter-fbgpj\" (UID: \"8f19f238-718d-44d6-86af-799a206bc26e\") " pod="openshift-network-operator/iptables-alerter-fbgpj"
Jul 31 14:48:24.469455 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-var-lib-openvswitch\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.469455 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-kubernetes\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.469455 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f19f238-718d-44d6-86af-799a206bc26e-host-slash\") pod \"iptables-alerter-fbgpj\" (UID: \"8f19f238-718d-44d6-86af-799a206bc26e\") " pod="openshift-network-operator/iptables-alerter-fbgpj"
Jul 31 14:48:24.469455 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-sysctl-d\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.469455 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-run\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.469455 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-sys\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.469455 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d391192c-cdbd-46ce-9098-444df02b6f1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.469455 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d391192c-cdbd-46ce-9098-444df02b6f1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.469455 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-slash\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.469455 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-hostroot\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.469455 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467823    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-modprobe-d\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.469455 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-var-lib-kubelet\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.469455 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/78ef4e44-0c0a-4875-9328-e83e1670ca8e-konnectivity-ca\") pod \"konnectivity-agent-f4ptw\" (UID: \"78ef4e44-0c0a-4875-9328-e83e1670ca8e\") " pod="kube-system/konnectivity-agent-f4ptw"
Jul 31 14:48:24.469455 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvp6q\" (UniqueName: \"kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q\") pod \"network-check-target-xs6tw\" (UID: \"4138e89c-a29b-4e92-8c6e-99b232125ef8\") " pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:24.469455 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c22c54bd-d71b-4169-8fc7-1810529a2e67-ovn-node-metrics-cert\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.469455 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-tuned\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.469455 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-cni-bin\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.470372 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.467998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c22c54bd-d71b-4169-8fc7-1810529a2e67-env-overrides\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.470372 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.468023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-cnibin\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.470372 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.468058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d391192c-cdbd-46ce-9098-444df02b6f1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.470372 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.468084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-multus-cni-dir\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.470372 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.468112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3961a7e-b75a-4837-944f-330a900f1b75-cni-binary-copy\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.470372 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.468139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2dlx\" (UniqueName: \"kubernetes.io/projected/352aa5e1-673c-4ede-a05a-80da4d81d2f9-kube-api-access-s2dlx\") pod \"network-metrics-daemon-97p2g\" (UID: \"352aa5e1-673c-4ede-a05a-80da4d81d2f9\") " pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:24.470372 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.468166    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6psvm\" (UniqueName: \"kubernetes.io/projected/2d63bb96-09d0-4751-b541-9e8095b24c58-kube-api-access-6psvm\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.470372 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.468212    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d391192c-cdbd-46ce-9098-444df02b6f1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.470372 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.468523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d391192c-cdbd-46ce-9098-444df02b6f1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.474376 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.474349    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 14:48:24.478355 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.478330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plmrw\" (UniqueName: \"kubernetes.io/projected/8f19f238-718d-44d6-86af-799a206bc26e-kube-api-access-plmrw\") pod \"iptables-alerter-fbgpj\" (UID: \"8f19f238-718d-44d6-86af-799a206bc26e\") " pod="openshift-network-operator/iptables-alerter-fbgpj"
Jul 31 14:48:24.478495 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.478412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q47j9\" (UniqueName: \"kubernetes.io/projected/d391192c-cdbd-46ce-9098-444df02b6f1b-kube-api-access-q47j9\") pod \"multus-additional-cni-plugins-4wc8q\" (UID: \"d391192c-cdbd-46ce-9098-444df02b6f1b\") " pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.478495 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.478412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bmdm\" (UniqueName: \"kubernetes.io/projected/fce1aee2-b141-4232-8fc0-d34fb333f976-kube-api-access-8bmdm\") pod \"aws-ebs-csi-driver-node-xghkk\" (UID: \"fce1aee2-b141-4232-8fc0-d34fb333f976\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.489875 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.489826    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal" event={"ID":"7cb07b957eab05242e711549788246d2","Type":"ContainerStarted","Data":"52baca642e76b303b1d185148e1b1d4099ea8bda94fed1d1e40fe010ed9220a8"}
Jul 31 14:48:24.491419 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.491392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal" event={"ID":"17290c901fea2fac12a7cbe3399d875c","Type":"ContainerStarted","Data":"45b00130a2c144fd29027a500c833b23806a8afadf050048074bfc2af36462ca"}
Jul 31 14:48:24.569312 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-cni-netd\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.569312 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-run-k8s-cni-cncf-io\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.569582 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569335    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-var-lib-cni-bin\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.569582 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d63bb96-09d0-4751-b541-9e8095b24c58-tmp\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.569582 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-multus-conf-dir\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.569582 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c3961a7e-b75a-4837-944f-330a900f1b75-multus-daemon-config\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.569582 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-kubelet-config\") pod \"global-pull-secret-syncer-kg6bm\" (UID: \"7fae56cb-8b98-4d8b-a508-62dac7aa73d1\") " pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:24.569582 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/82144e0f-aab6-4481-ab7f-86a85e384b20-serviceca\") pod \"node-ca-f74pb\" (UID: \"82144e0f-aab6-4481-ab7f-86a85e384b20\") " pod="openshift-image-registry/node-ca-f74pb"
Jul 31 14:48:24.569582 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-run-ovn\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.569582 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg282\" (UniqueName: \"kubernetes.io/projected/c22c54bd-d71b-4169-8fc7-1810529a2e67-kube-api-access-mg282\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.569582 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-var-lib-kubelet\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.569582 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxfdz\" (UniqueName: \"kubernetes.io/projected/c3961a7e-b75a-4837-944f-330a900f1b75-kube-api-access-wxfdz\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.569582 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-systemd\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.569582 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-etc-openvswitch\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.570196 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-run-ovn-kubernetes\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.570196 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c22c54bd-d71b-4169-8fc7-1810529a2e67-ovnkube-config\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.570196 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569637    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-dbus\") pod \"global-pull-secret-syncer-kg6bm\" (UID: \"7fae56cb-8b98-4d8b-a508-62dac7aa73d1\") " pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:24.570196 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-lib-modules\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.570196 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-host\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.570196 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569707    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret\") pod \"global-pull-secret-syncer-kg6bm\" (UID: \"7fae56cb-8b98-4d8b-a508-62dac7aa73d1\") " pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:24.570196 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569775    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-run-ovn\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.570196 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:24.569829    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:24.570196 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-multus-socket-dir-parent\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.570196 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82144e0f-aab6-4481-ab7f-86a85e384b20-host\") pod \"node-ca-f74pb\" (UID: \"82144e0f-aab6-4481-ab7f-86a85e384b20\") " pod="openshift-image-registry/node-ca-f74pb"
Jul 31 14:48:24.570196 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569890    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-etc-openvswitch\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.570196 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82144e0f-aab6-4481-ab7f-86a85e384b20-host\") pod \"node-ca-f74pb\" (UID: \"82144e0f-aab6-4481-ab7f-86a85e384b20\") " pod="openshift-image-registry/node-ca-f74pb"
Jul 31 14:48:24.570196 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.569959    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-var-lib-kubelet\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.570196 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-dbus\") pod \"global-pull-secret-syncer-kg6bm\" (UID: \"7fae56cb-8b98-4d8b-a508-62dac7aa73d1\") " pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:24.570196 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570111    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-var-lib-cni-bin\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.570196 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570162    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-run-ovn-kubernetes\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.570196 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-host\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.571086 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570304    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-lib-modules\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.571086 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570314    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-systemd\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.571086 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570396    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-run-k8s-cni-cncf-io\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.571086 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-kubelet-config\") pod \"global-pull-secret-syncer-kg6bm\" (UID: \"7fae56cb-8b98-4d8b-a508-62dac7aa73d1\") " pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:24.571086 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-multus-socket-dir-parent\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.571086 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:24.569923    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret podName:7fae56cb-8b98-4d8b-a508-62dac7aa73d1 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:25.069881367 +0000 UTC m=+3.079093231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret") pod "global-pull-secret-syncer-kg6bm" (UID: "7fae56cb-8b98-4d8b-a508-62dac7aa73d1") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:24.571086 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-multus-conf-dir\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.571086 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570482    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-cni-netd\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.571086 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570582    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c3961a7e-b75a-4837-944f-330a900f1b75-multus-daemon-config\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.571086 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-run-systemd\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.571086 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570637    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-var-lib-cni-multus\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.571086 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570650    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-run-systemd\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.571086 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-sysconfig\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.571086 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-var-lib-openvswitch\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.571086 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-kubernetes\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.571086 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570708    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-var-lib-cni-multus\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.571086 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-sysctl-d\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-run\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-var-lib-openvswitch\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570787    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-kubernetes\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570802    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-sysconfig\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570801    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c22c54bd-d71b-4169-8fc7-1810529a2e67-ovnkube-config\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-sys\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570890    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/82144e0f-aab6-4481-ab7f-86a85e384b20-serviceca\") pod \"node-ca-f74pb\" (UID: \"82144e0f-aab6-4481-ab7f-86a85e384b20\") " pod="openshift-image-registry/node-ca-f74pb"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-run\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-slash\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570832    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-sysctl-d\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-hostroot\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-sys\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-modprobe-d\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-var-lib-kubelet\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/78ef4e44-0c0a-4875-9328-e83e1670ca8e-konnectivity-ca\") pod \"konnectivity-agent-f4ptw\" (UID: \"78ef4e44-0c0a-4875-9328-e83e1670ca8e\") " pod="kube-system/konnectivity-agent-f4ptw"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-hostroot\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvp6q\" (UniqueName: \"kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q\") pod \"network-check-target-xs6tw\" (UID: \"4138e89c-a29b-4e92-8c6e-99b232125ef8\") " pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:24.572030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c22c54bd-d71b-4169-8fc7-1810529a2e67-ovn-node-metrics-cert\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-tuned\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-cni-bin\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571083    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-modprobe-d\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c22c54bd-d71b-4169-8fc7-1810529a2e67-env-overrides\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-cnibin\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571139    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-multus-cni-dir\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571140    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-var-lib-kubelet\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3961a7e-b75a-4837-944f-330a900f1b75-cni-binary-copy\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2dlx\" (UniqueName: \"kubernetes.io/projected/352aa5e1-673c-4ede-a05a-80da4d81d2f9-kube-api-access-s2dlx\") pod \"network-metrics-daemon-97p2g\" (UID: \"352aa5e1-673c-4ede-a05a-80da4d81d2f9\") " pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-cni-bin\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.570993    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-slash\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/78ef4e44-0c0a-4875-9328-e83e1670ca8e-konnectivity-ca\") pod \"konnectivity-agent-f4ptw\" (UID: \"78ef4e44-0c0a-4875-9328-e83e1670ca8e\") " pod="kube-system/konnectivity-agent-f4ptw"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571646    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-multus-cni-dir\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571703    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-cnibin\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6psvm\" (UniqueName: \"kubernetes.io/projected/2d63bb96-09d0-4751-b541-9e8095b24c58-kube-api-access-6psvm\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx4hj\" (UniqueName: \"kubernetes.io/projected/be585c19-1ea7-4e53-8142-da3d9b2bc15a-kube-api-access-cx4hj\") pod \"node-resolver-x2h67\" (UID: \"be585c19-1ea7-4e53-8142-da3d9b2bc15a\") " pod="openshift-dns/node-resolver-x2h67"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp78b\" (UniqueName: \"kubernetes.io/projected/82144e0f-aab6-4481-ab7f-86a85e384b20-kube-api-access-vp78b\") pod \"node-ca-f74pb\" (UID: \"82144e0f-aab6-4481-ab7f-86a85e384b20\") " pod="openshift-image-registry/node-ca-f74pb"
Jul 31 14:48:24.572956 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-run-openvswitch\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-etc-kubernetes\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/78ef4e44-0c0a-4875-9328-e83e1670ca8e-agent-certs\") pod \"konnectivity-agent-f4ptw\" (UID: \"78ef4e44-0c0a-4875-9328-e83e1670ca8e\") " pod="kube-system/konnectivity-agent-f4ptw"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be585c19-1ea7-4e53-8142-da3d9b2bc15a-hosts-file\") pod \"node-resolver-x2h67\" (UID: \"be585c19-1ea7-4e53-8142-da3d9b2bc15a\") " pod="openshift-dns/node-resolver-x2h67"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.571984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-kubelet\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.572010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-run-netns\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.572035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-sysctl-conf\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.572062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be585c19-1ea7-4e53-8142-da3d9b2bc15a-tmp-dir\") pod \"node-resolver-x2h67\" (UID: \"be585c19-1ea7-4e53-8142-da3d9b2bc15a\") " pod="openshift-dns/node-resolver-x2h67"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.572092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-system-cni-dir\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.572119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-os-release\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.572147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-run-netns\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.572173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-run-multus-certs\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.572232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs\") pod \"network-metrics-daemon-97p2g\" (UID: \"352aa5e1-673c-4ede-a05a-80da4d81d2f9\") " pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.572260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-systemd-units\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.572287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c22c54bd-d71b-4169-8fc7-1810529a2e67-ovnkube-script-lib\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.572317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-node-log\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.572342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-log-socket\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.573846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.572467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c22c54bd-d71b-4169-8fc7-1810529a2e67-env-overrides\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.572474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-log-socket\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.572762    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-run-openvswitch\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.572811    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.572853    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-etc-kubernetes\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:24.573264    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.573271    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-os-release\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:24.573333    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs podName:352aa5e1-673c-4ede-a05a-80da4d81d2f9 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:25.07331357 +0000 UTC m=+3.082525435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs") pod "network-metrics-daemon-97p2g" (UID: "352aa5e1-673c-4ede-a05a-80da4d81d2f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.573330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/be585c19-1ea7-4e53-8142-da3d9b2bc15a-hosts-file\") pod \"node-resolver-x2h67\" (UID: \"be585c19-1ea7-4e53-8142-da3d9b2bc15a\") " pod="openshift-dns/node-resolver-x2h67"
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.573369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-kubelet\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.573384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-run-netns\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.573448    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-host-run-netns\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.573502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-host-run-multus-certs\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.573545    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3961a7e-b75a-4837-944f-330a900f1b75-cni-binary-copy\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.573622    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3961a7e-b75a-4837-944f-330a900f1b75-system-cni-dir\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.573632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-sysctl-conf\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.573693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-systemd-units\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.573696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c22c54bd-d71b-4169-8fc7-1810529a2e67-node-log\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.574798 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.573886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2d63bb96-09d0-4751-b541-9e8095b24c58-etc-tuned\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.575801 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.574011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c22c54bd-d71b-4169-8fc7-1810529a2e67-ovnkube-script-lib\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.575801 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.574018    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be585c19-1ea7-4e53-8142-da3d9b2bc15a-tmp-dir\") pod \"node-resolver-x2h67\" (UID: \"be585c19-1ea7-4e53-8142-da3d9b2bc15a\") " pod="openshift-dns/node-resolver-x2h67"
Jul 31 14:48:24.575801 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.574678    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c22c54bd-d71b-4169-8fc7-1810529a2e67-ovn-node-metrics-cert\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.576908 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.576229    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/78ef4e44-0c0a-4875-9328-e83e1670ca8e-agent-certs\") pod \"konnectivity-agent-f4ptw\" (UID: \"78ef4e44-0c0a-4875-9328-e83e1670ca8e\") " pod="kube-system/konnectivity-agent-f4ptw"
Jul 31 14:48:24.578262 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.578238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d63bb96-09d0-4751-b541-9e8095b24c58-tmp\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.594860 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:24.594784    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:24.594860 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:24.594814    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:24.594860 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:24.594829    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pvp6q for pod openshift-network-diagnostics/network-check-target-xs6tw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:24.595143 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:24.594895    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q podName:4138e89c-a29b-4e92-8c6e-99b232125ef8 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:25.094877165 +0000 UTC m=+3.104089021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pvp6q" (UniqueName: "kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q") pod "network-check-target-xs6tw" (UID: "4138e89c-a29b-4e92-8c6e-99b232125ef8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:24.596772 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.596746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxfdz\" (UniqueName: \"kubernetes.io/projected/c3961a7e-b75a-4837-944f-330a900f1b75-kube-api-access-wxfdz\") pod \"multus-sxt8b\" (UID: \"c3961a7e-b75a-4837-944f-330a900f1b75\") " pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.597485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.596913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx4hj\" (UniqueName: \"kubernetes.io/projected/be585c19-1ea7-4e53-8142-da3d9b2bc15a-kube-api-access-cx4hj\") pod \"node-resolver-x2h67\" (UID: \"be585c19-1ea7-4e53-8142-da3d9b2bc15a\") " pod="openshift-dns/node-resolver-x2h67"
Jul 31 14:48:24.597485 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.597378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp78b\" (UniqueName: \"kubernetes.io/projected/82144e0f-aab6-4481-ab7f-86a85e384b20-kube-api-access-vp78b\") pod \"node-ca-f74pb\" (UID: \"82144e0f-aab6-4481-ab7f-86a85e384b20\") " pod="openshift-image-registry/node-ca-f74pb"
Jul 31 14:48:24.597749 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.597730    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6psvm\" (UniqueName: \"kubernetes.io/projected/2d63bb96-09d0-4751-b541-9e8095b24c58-kube-api-access-6psvm\") pod \"tuned-x4rzq\" (UID: \"2d63bb96-09d0-4751-b541-9e8095b24c58\") " pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.598499 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.598474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2dlx\" (UniqueName: \"kubernetes.io/projected/352aa5e1-673c-4ede-a05a-80da4d81d2f9-kube-api-access-s2dlx\") pod \"network-metrics-daemon-97p2g\" (UID: \"352aa5e1-673c-4ede-a05a-80da4d81d2f9\") " pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:24.598878 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.598854    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg282\" (UniqueName: \"kubernetes.io/projected/c22c54bd-d71b-4169-8fc7-1810529a2e67-kube-api-access-mg282\") pod \"ovnkube-node-lhnwk\" (UID: \"c22c54bd-d71b-4169-8fc7-1810529a2e67\") " pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.663036 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.663000    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk"
Jul 31 14:48:24.670822 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.670791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4wc8q"
Jul 31 14:48:24.679335 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.679311    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fbgpj"
Jul 31 14:48:24.686538 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.686345    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sxt8b"
Jul 31 14:48:24.691331 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.691303    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f4ptw"
Jul 31 14:48:24.700184 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.700155    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x4rzq"
Jul 31 14:48:24.706188 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.706162    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x2h67"
Jul 31 14:48:24.712930 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.712911    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f74pb"
Jul 31 14:48:24.717915 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.717893    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:24.878562 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:24.878478    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:25.076952 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:25.076917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret\") pod \"global-pull-secret-syncer-kg6bm\" (UID: \"7fae56cb-8b98-4d8b-a508-62dac7aa73d1\") " pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:25.077141 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:25.077005    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs\") pod \"network-metrics-daemon-97p2g\" (UID: \"352aa5e1-673c-4ede-a05a-80da4d81d2f9\") " pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:25.077141 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:25.077090    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:25.077141 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:25.077120    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:25.077305 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:25.077175    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret podName:7fae56cb-8b98-4d8b-a508-62dac7aa73d1 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:26.077154803 +0000 UTC m=+4.086366676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret") pod "global-pull-secret-syncer-kg6bm" (UID: "7fae56cb-8b98-4d8b-a508-62dac7aa73d1") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:25.077305 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:25.077192    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs podName:352aa5e1-673c-4ede-a05a-80da4d81d2f9 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:26.077185128 +0000 UTC m=+4.086396978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs") pod "network-metrics-daemon-97p2g" (UID: "352aa5e1-673c-4ede-a05a-80da4d81d2f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:25.177415 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:25.177305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvp6q\" (UniqueName: \"kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q\") pod \"network-check-target-xs6tw\" (UID: \"4138e89c-a29b-4e92-8c6e-99b232125ef8\") " pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:25.177590 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:25.177507    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:25.177590 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:25.177541    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:25.177590 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:25.177555    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pvp6q for pod openshift-network-diagnostics/network-check-target-xs6tw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:25.177759 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:25.177630    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q podName:4138e89c-a29b-4e92-8c6e-99b232125ef8 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:26.17760923 +0000 UTC m=+4.186821084 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pvp6q" (UniqueName: "kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q") pod "network-check-target-xs6tw" (UID: "4138e89c-a29b-4e92-8c6e-99b232125ef8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:25.404027 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:25.403971    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:43:25 +0000 UTC" deadline="2028-03-02 04:58:59.854106418 +0000 UTC"
Jul 31 14:48:25.404027 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:25.404019    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13910h10m34.450090426s"
Jul 31 14:48:25.437147 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:25.437071    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:25.485064 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:25.485030    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:25.485215 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:25.485172    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:25.702134 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:25.702104    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ef4e44_0c0a_4875_9328_e83e1670ca8e.slice/crio-f5b0e585929cfeed5f777e00664158e2ef2859c620a379ae0c5d2c3572ad12e2 WatchSource:0}: Error finding container f5b0e585929cfeed5f777e00664158e2ef2859c620a379ae0c5d2c3572ad12e2: Status 404 returned error can't find the container with id f5b0e585929cfeed5f777e00664158e2ef2859c620a379ae0c5d2c3572ad12e2
Jul 31 14:48:25.703070 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:25.703001    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd391192c_cdbd_46ce_9098_444df02b6f1b.slice/crio-462af4edd529782b9c55fe4b1a0c3c408d7254b8bb282f6eaed83a4f8c13a944 WatchSource:0}: Error finding container 462af4edd529782b9c55fe4b1a0c3c408d7254b8bb282f6eaed83a4f8c13a944: Status 404 returned error can't find the container with id 462af4edd529782b9c55fe4b1a0c3c408d7254b8bb282f6eaed83a4f8c13a944
Jul 31 14:48:25.708211 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:25.708165    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce1aee2_b141_4232_8fc0_d34fb333f976.slice/crio-8e5b388cf0902306e6ca18051a9144c0fb50f57c2faa1a939efdb2182da0fd1c WatchSource:0}: Error finding container 8e5b388cf0902306e6ca18051a9144c0fb50f57c2faa1a939efdb2182da0fd1c: Status 404 returned error can't find the container with id 8e5b388cf0902306e6ca18051a9144c0fb50f57c2faa1a939efdb2182da0fd1c
Jul 31 14:48:25.709890 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:25.709866    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f19f238_718d_44d6_86af_799a206bc26e.slice/crio-87fc9ac7c165431600ab3a55d7f0d8284792a27841e2a04ac93fe06d84d1d77d WatchSource:0}: Error finding container 87fc9ac7c165431600ab3a55d7f0d8284792a27841e2a04ac93fe06d84d1d77d: Status 404 returned error can't find the container with id 87fc9ac7c165431600ab3a55d7f0d8284792a27841e2a04ac93fe06d84d1d77d
Jul 31 14:48:25.710832 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:25.710806    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82144e0f_aab6_4481_ab7f_86a85e384b20.slice/crio-00e7f22e8a9bf00ce50c8cc56dcba92523f98abaf5f7734536f2eca9a191d461 WatchSource:0}: Error finding container 00e7f22e8a9bf00ce50c8cc56dcba92523f98abaf5f7734536f2eca9a191d461: Status 404 returned error can't find the container with id 00e7f22e8a9bf00ce50c8cc56dcba92523f98abaf5f7734536f2eca9a191d461
Jul 31 14:48:25.711182 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:25.711112    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d63bb96_09d0_4751_b541_9e8095b24c58.slice/crio-97fb12c8151d5eadd8eb67ee2170d205c797c0163f0fc9451ce12acbfbb8f3b0 WatchSource:0}: Error finding container 97fb12c8151d5eadd8eb67ee2170d205c797c0163f0fc9451ce12acbfbb8f3b0: Status 404 returned error can't find the container with id 97fb12c8151d5eadd8eb67ee2170d205c797c0163f0fc9451ce12acbfbb8f3b0
Jul 31 14:48:25.717475 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:25.717448    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3961a7e_b75a_4837_944f_330a900f1b75.slice/crio-6f3d891312aa2a4f857c30024f0412688f543dc283d81f116dc47b6192ee6238 WatchSource:0}: Error finding container 6f3d891312aa2a4f857c30024f0412688f543dc283d81f116dc47b6192ee6238: Status 404 returned error can't find the container with id 6f3d891312aa2a4f857c30024f0412688f543dc283d81f116dc47b6192ee6238
Jul 31 14:48:26.084334 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.084029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs\") pod \"network-metrics-daemon-97p2g\" (UID: \"352aa5e1-673c-4ede-a05a-80da4d81d2f9\") " pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:26.084514 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.084338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret\") pod \"global-pull-secret-syncer-kg6bm\" (UID: \"7fae56cb-8b98-4d8b-a508-62dac7aa73d1\") " pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:26.084514 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:26.084202    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:26.084514 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:26.084475    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs podName:352aa5e1-673c-4ede-a05a-80da4d81d2f9 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:28.084451016 +0000 UTC m=+6.093662884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs") pod "network-metrics-daemon-97p2g" (UID: "352aa5e1-673c-4ede-a05a-80da4d81d2f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:26.084514 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:26.084489    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:26.084775 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:26.084543    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret podName:7fae56cb-8b98-4d8b-a508-62dac7aa73d1 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:28.084528122 +0000 UTC m=+6.093739987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret") pod "global-pull-secret-syncer-kg6bm" (UID: "7fae56cb-8b98-4d8b-a508-62dac7aa73d1") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:26.186027 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.185929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvp6q\" (UniqueName: \"kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q\") pod \"network-check-target-xs6tw\" (UID: \"4138e89c-a29b-4e92-8c6e-99b232125ef8\") " pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:26.186211 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:26.186118    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:26.186211 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:26.186147    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:26.186211 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:26.186161    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pvp6q for pod openshift-network-diagnostics/network-check-target-xs6tw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:26.186406 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:26.186227    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q podName:4138e89c-a29b-4e92-8c6e-99b232125ef8 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:28.186207836 +0000 UTC m=+6.195419700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pvp6q" (UniqueName: "kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q") pod "network-check-target-xs6tw" (UID: "4138e89c-a29b-4e92-8c6e-99b232125ef8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:26.404995 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.404891    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:43:25 +0000 UTC" deadline="2028-04-10 19:49:09.440668676 +0000 UTC"
Jul 31 14:48:26.404995 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.404934    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14861h0m43.035739717s"
Jul 31 14:48:26.493517 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.492850    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:26.493517 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:26.493007    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:26.494101 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.493929    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:26.494101 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:26.494057    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:26.519003 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.518499    2578 generic.go:356] "Generic (PLEG): container finished" podID="7cb07b957eab05242e711549788246d2" containerID="dcfae400411eedb2fb140bb7521f57b2835a09255aa4a5b72a8bbba007a34a56" exitCode=0
Jul 31 14:48:26.519448 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.519228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal" event={"ID":"7cb07b957eab05242e711549788246d2","Type":"ContainerDied","Data":"dcfae400411eedb2fb140bb7521f57b2835a09255aa4a5b72a8bbba007a34a56"}
Jul 31 14:48:26.534897 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.534826    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal" event={"ID":"17290c901fea2fac12a7cbe3399d875c","Type":"ContainerStarted","Data":"a1399c1c1e6f5297be31663083267dc89d301ffce2e5653dcbbd1afe2a7fda17"}
Jul 31 14:48:26.545486 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.545373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk" event={"ID":"c22c54bd-d71b-4169-8fc7-1810529a2e67","Type":"ContainerStarted","Data":"cfff657fa590d1bd2f4078cd12222f10b93fed6d87b4f077c339735bc8673774"}
Jul 31 14:48:26.558189 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.558156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x2h67" event={"ID":"be585c19-1ea7-4e53-8142-da3d9b2bc15a","Type":"ContainerStarted","Data":"302517c5f27993bfd767d34285c46c28a7275de892ec0c0219a3c267568a95f2"}
Jul 31 14:48:26.573894 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.573864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x4rzq" event={"ID":"2d63bb96-09d0-4751-b541-9e8095b24c58","Type":"ContainerStarted","Data":"97fb12c8151d5eadd8eb67ee2170d205c797c0163f0fc9451ce12acbfbb8f3b0"}
Jul 31 14:48:26.588685 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.588613    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fbgpj" event={"ID":"8f19f238-718d-44d6-86af-799a206bc26e","Type":"ContainerStarted","Data":"87fc9ac7c165431600ab3a55d7f0d8284792a27841e2a04ac93fe06d84d1d77d"}
Jul 31 14:48:26.590036 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.590011    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk" event={"ID":"fce1aee2-b141-4232-8fc0-d34fb333f976","Type":"ContainerStarted","Data":"8e5b388cf0902306e6ca18051a9144c0fb50f57c2faa1a939efdb2182da0fd1c"}
Jul 31 14:48:26.599930 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.599904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sxt8b" event={"ID":"c3961a7e-b75a-4837-944f-330a900f1b75","Type":"ContainerStarted","Data":"6f3d891312aa2a4f857c30024f0412688f543dc283d81f116dc47b6192ee6238"}
Jul 31 14:48:26.618982 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.618927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f74pb" event={"ID":"82144e0f-aab6-4481-ab7f-86a85e384b20","Type":"ContainerStarted","Data":"00e7f22e8a9bf00ce50c8cc56dcba92523f98abaf5f7734536f2eca9a191d461"}
Jul 31 14:48:26.638712 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.638680    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wc8q" event={"ID":"d391192c-cdbd-46ce-9098-444df02b6f1b","Type":"ContainerStarted","Data":"462af4edd529782b9c55fe4b1a0c3c408d7254b8bb282f6eaed83a4f8c13a944"}
Jul 31 14:48:26.651840 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:26.651776    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f4ptw" event={"ID":"78ef4e44-0c0a-4875-9328-e83e1670ca8e","Type":"ContainerStarted","Data":"f5b0e585929cfeed5f777e00664158e2ef2859c620a379ae0c5d2c3572ad12e2"}
Jul 31 14:48:27.485702 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:27.485146    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:27.485702 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:27.485310    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:27.666778 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:27.666676    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal" event={"ID":"7cb07b957eab05242e711549788246d2","Type":"ContainerStarted","Data":"630ed28e3c6584884ef4478b8b5482e17ddf027c84a8b95afd04978747ec6537"}
Jul 31 14:48:27.700463 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:27.699487    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal" podStartSLOduration=3.69946117 podStartE2EDuration="3.69946117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:26.561334938 +0000 UTC m=+4.570546812" watchObservedRunningTime="2026-07-31 14:48:27.69946117 +0000 UTC m=+5.708673044"
Jul 31 14:48:28.108542 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:28.108500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret\") pod \"global-pull-secret-syncer-kg6bm\" (UID: \"7fae56cb-8b98-4d8b-a508-62dac7aa73d1\") " pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:28.108720 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:28.108601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs\") pod \"network-metrics-daemon-97p2g\" (UID: \"352aa5e1-673c-4ede-a05a-80da4d81d2f9\") " pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:28.108780 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:28.108742    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:28.108849 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:28.108809    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs podName:352aa5e1-673c-4ede-a05a-80da4d81d2f9 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:32.108788796 +0000 UTC m=+10.118000659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs") pod "network-metrics-daemon-97p2g" (UID: "352aa5e1-673c-4ede-a05a-80da4d81d2f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:28.109252 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:28.109234    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:28.109338 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:28.109292    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret podName:7fae56cb-8b98-4d8b-a508-62dac7aa73d1 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:32.109275751 +0000 UTC m=+10.118487616 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret") pod "global-pull-secret-syncer-kg6bm" (UID: "7fae56cb-8b98-4d8b-a508-62dac7aa73d1") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:28.209833 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:28.209787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvp6q\" (UniqueName: \"kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q\") pod \"network-check-target-xs6tw\" (UID: \"4138e89c-a29b-4e92-8c6e-99b232125ef8\") " pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:28.210111 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:28.210029    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:28.210111 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:28.210054    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:28.210111 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:28.210068    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pvp6q for pod openshift-network-diagnostics/network-check-target-xs6tw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:28.210288 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:28.210147    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q podName:4138e89c-a29b-4e92-8c6e-99b232125ef8 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:32.210126494 +0000 UTC m=+10.219338367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pvp6q" (UniqueName: "kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q") pod "network-check-target-xs6tw" (UID: "4138e89c-a29b-4e92-8c6e-99b232125ef8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:28.487645 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:28.487566    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:28.488112 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:28.487691    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:28.488112 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:28.488094    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:28.488232 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:28.488196    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:29.484234 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:29.484194    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:29.484520 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:29.484340    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:30.485678 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:30.485075    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:30.485678 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:30.485253    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:30.485678 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:30.485540    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:30.486263 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:30.485696    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:31.484971 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:31.484919    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:31.485168 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:31.485085    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:32.142803 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:32.142748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret\") pod \"global-pull-secret-syncer-kg6bm\" (UID: \"7fae56cb-8b98-4d8b-a508-62dac7aa73d1\") " pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:32.143375 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:32.142855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs\") pod \"network-metrics-daemon-97p2g\" (UID: \"352aa5e1-673c-4ede-a05a-80da4d81d2f9\") " pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:32.143375 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:32.142993    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:32.143375 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:32.143065    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs podName:352aa5e1-673c-4ede-a05a-80da4d81d2f9 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:40.143043723 +0000 UTC m=+18.152255807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs") pod "network-metrics-daemon-97p2g" (UID: "352aa5e1-673c-4ede-a05a-80da4d81d2f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:32.143589 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:32.143531    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:32.143650 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:32.143592    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret podName:7fae56cb-8b98-4d8b-a508-62dac7aa73d1 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:40.14357379 +0000 UTC m=+18.152785662 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret") pod "global-pull-secret-syncer-kg6bm" (UID: "7fae56cb-8b98-4d8b-a508-62dac7aa73d1") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:32.244022 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:32.243955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvp6q\" (UniqueName: \"kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q\") pod \"network-check-target-xs6tw\" (UID: \"4138e89c-a29b-4e92-8c6e-99b232125ef8\") " pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:32.244235 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:32.244185    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:32.244235 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:32.244216    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:32.244235 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:32.244231    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pvp6q for pod openshift-network-diagnostics/network-check-target-xs6tw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:32.244434 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:32.244307    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q podName:4138e89c-a29b-4e92-8c6e-99b232125ef8 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:40.244286433 +0000 UTC m=+18.253498288 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pvp6q" (UniqueName: "kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q") pod "network-check-target-xs6tw" (UID: "4138e89c-a29b-4e92-8c6e-99b232125ef8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:32.486958 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:32.486878    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:32.488618 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:32.488035    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:32.488618 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:32.488461    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:32.488618 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:32.488568    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:33.485042 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:33.484995    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:33.485631 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:33.485163    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:34.484838 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:34.484799    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:34.485010 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:34.484934    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:34.485380 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:34.485341    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:34.485783 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:34.485454    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:35.484160 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:35.484122    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:35.484382 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:35.484269    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:36.485203 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:36.485169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:36.485677 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:36.485299    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:36.485677 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:36.485366    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:36.485677 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:36.485476    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:37.485145 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:37.485112    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:37.485324 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:37.485268    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:38.484772 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:38.484734    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:38.484954 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:38.484741    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:38.484954 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:38.484884    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:38.484954 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:38.484941    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:39.484992 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:39.484959    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:39.485496 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:39.485086    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:40.201610 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:40.201562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret\") pod \"global-pull-secret-syncer-kg6bm\" (UID: \"7fae56cb-8b98-4d8b-a508-62dac7aa73d1\") " pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:40.201805 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:40.201736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs\") pod \"network-metrics-daemon-97p2g\" (UID: \"352aa5e1-673c-4ede-a05a-80da4d81d2f9\") " pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:40.201805 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:40.201756    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:40.201914 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:40.201831    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:40.201914 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:40.201843    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret podName:7fae56cb-8b98-4d8b-a508-62dac7aa73d1 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:56.201821852 +0000 UTC m=+34.211033720 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret") pod "global-pull-secret-syncer-kg6bm" (UID: "7fae56cb-8b98-4d8b-a508-62dac7aa73d1") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:40.201914 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:40.201882    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs podName:352aa5e1-673c-4ede-a05a-80da4d81d2f9 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:56.201866073 +0000 UTC m=+34.211077940 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs") pod "network-metrics-daemon-97p2g" (UID: "352aa5e1-673c-4ede-a05a-80da4d81d2f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:40.302665 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:40.302624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvp6q\" (UniqueName: \"kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q\") pod \"network-check-target-xs6tw\" (UID: \"4138e89c-a29b-4e92-8c6e-99b232125ef8\") " pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:40.302863 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:40.302790    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:40.302863 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:40.302812    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:40.302863 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:40.302826    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pvp6q for pod openshift-network-diagnostics/network-check-target-xs6tw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:40.303031 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:40.302891    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q podName:4138e89c-a29b-4e92-8c6e-99b232125ef8 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:56.30287195 +0000 UTC m=+34.312083800 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pvp6q" (UniqueName: "kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q") pod "network-check-target-xs6tw" (UID: "4138e89c-a29b-4e92-8c6e-99b232125ef8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:40.484406 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:40.484326    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:40.484553 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:40.484339    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:40.484553 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:40.484474    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:40.484666 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:40.484544    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:41.484689 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:41.484656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:41.485143 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:41.484776    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:42.485801 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:42.485767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:42.486213 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:42.485871    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:42.486213 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:42.485952    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:42.486213 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:42.486047    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:43.485141 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:43.485106    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:43.485294 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:43.485237    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:44.487704 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:44.487671    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:44.488122 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:44.487677    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:44.488122 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:44.487799    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:44.488122 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:44.487852    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:45.485160 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.484869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:45.485444 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:45.485239    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:45.705807 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.705727    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk" event={"ID":"fce1aee2-b141-4232-8fc0-d34fb333f976","Type":"ContainerStarted","Data":"bddde8ed2edcaad4594f1485fe6198c7f208bc079966aadfc85e25d4bdc4da82"}
Jul 31 14:48:45.706982 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.706961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sxt8b" event={"ID":"c3961a7e-b75a-4837-944f-330a900f1b75","Type":"ContainerStarted","Data":"9859b4f0463151632bfd7d312b84a091db15b1e0b0c03f2cc0a5dc13aa515cc7"}
Jul 31 14:48:45.708338 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.708314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f74pb" event={"ID":"82144e0f-aab6-4481-ab7f-86a85e384b20","Type":"ContainerStarted","Data":"734afbe873aa393d262c04dba45de6194450bf5a4fe0f624f1f9caa32d70afce"}
Jul 31 14:48:45.709568 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.709549    2578 generic.go:356] "Generic (PLEG): container finished" podID="d391192c-cdbd-46ce-9098-444df02b6f1b" containerID="c72725c9e9d8b74f9d347fdd49cc84d02242c4f78657b39baeb55dde95082f99" exitCode=0
Jul 31 14:48:45.709649 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.709608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wc8q" event={"ID":"d391192c-cdbd-46ce-9098-444df02b6f1b","Type":"ContainerDied","Data":"c72725c9e9d8b74f9d347fdd49cc84d02242c4f78657b39baeb55dde95082f99"}
Jul 31 14:48:45.711114 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.711088    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f4ptw" event={"ID":"78ef4e44-0c0a-4875-9328-e83e1670ca8e","Type":"ContainerStarted","Data":"298745c163bf7a4253d6566e31af4b3be449bde4ebcc3ae02a9187364761e10c"}
Jul 31 14:48:45.713931 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.713907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk" event={"ID":"c22c54bd-d71b-4169-8fc7-1810529a2e67","Type":"ContainerStarted","Data":"fe56e4801bd34a17c6700fa4f07deccc1934f0125b70ac95e43bb1af33e4c7e3"}
Jul 31 14:48:45.714008 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.713965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk" event={"ID":"c22c54bd-d71b-4169-8fc7-1810529a2e67","Type":"ContainerStarted","Data":"e06c065fb7046fac885ca07771e2ef5ef8f27fcd996b34a66a65bff5ef57bfa3"}
Jul 31 14:48:45.714008 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.714005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk" event={"ID":"c22c54bd-d71b-4169-8fc7-1810529a2e67","Type":"ContainerStarted","Data":"8b066bac4cca371be604b0b672d11f86fe8ae7c3d7873660f933ddaf3c7f9bdb"}
Jul 31 14:48:45.714089 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.714041    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk" event={"ID":"c22c54bd-d71b-4169-8fc7-1810529a2e67","Type":"ContainerStarted","Data":"8daa9fc32561ddcbf376a651c9c554f43543c6b60e2eedccbff3013b47c7785a"}
Jul 31 14:48:45.714089 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.714079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk" event={"ID":"c22c54bd-d71b-4169-8fc7-1810529a2e67","Type":"ContainerStarted","Data":"2dbf9fda0ee248bfb339220d4da9424635dd459aef96b86bf227a223833c8600"}
Jul 31 14:48:45.714184 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.714117    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk" event={"ID":"c22c54bd-d71b-4169-8fc7-1810529a2e67","Type":"ContainerStarted","Data":"aa0d541986a48b5b711fb06237057f009af455a16efa9aa6537498474d5216ba"}
Jul 31 14:48:45.715169 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.715148    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x2h67" event={"ID":"be585c19-1ea7-4e53-8142-da3d9b2bc15a","Type":"ContainerStarted","Data":"00681af8bfb1495287efb0468fbb1492c87b0c422e5721f7c1c2848cdd7fed69"}
Jul 31 14:48:45.716817 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.716796    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x4rzq" event={"ID":"2d63bb96-09d0-4751-b541-9e8095b24c58","Type":"ContainerStarted","Data":"1ed7e776ec7cce467eb0cf7c6b6ffd3535f2d7e80b644909875911d38eb45956"}
Jul 31 14:48:45.728100 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.728066    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal" podStartSLOduration=21.728049061 podStartE2EDuration="21.728049061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:27.701622196 +0000 UTC m=+5.710834071" watchObservedRunningTime="2026-07-31 14:48:45.728049061 +0000 UTC m=+23.737260929"
Jul 31 14:48:45.728228 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.728210    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sxt8b" podStartSLOduration=3.655209065 podStartE2EDuration="22.728203759s" totalImagesPullingTime="19.072994694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:45.727829828 +0000 UTC m=+23.737041697" watchObservedRunningTime="2026-07-31 14:48:45.728203759 +0000 UTC m=+23.737415628"
Jul 31 14:48:45.773059 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.773020    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f4ptw" podStartSLOduration=3.7419286510000003 podStartE2EDuration="22.77300696s" totalImagesPullingTime="19.031078309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:45.772609857 +0000 UTC m=+23.781821726" watchObservedRunningTime="2026-07-31 14:48:45.77300696 +0000 UTC m=+23.782218828"
Jul 31 14:48:45.797008 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.796957    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-x4rzq" podStartSLOduration=3.772973727 podStartE2EDuration="22.796942297s" totalImagesPullingTime="19.02396857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:45.796604462 +0000 UTC m=+23.805816331" watchObservedRunningTime="2026-07-31 14:48:45.796942297 +0000 UTC m=+23.806154165"
Jul 31 14:48:45.817406 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.817342    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f74pb" podStartSLOduration=11.551530006 podStartE2EDuration="22.817330283s" totalImagesPullingTime="11.265800277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:45.817135715 +0000 UTC m=+23.826347584" watchObservedRunningTime="2026-07-31 14:48:45.817330283 +0000 UTC m=+23.826542151"
Jul 31 14:48:45.845368 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:45.845315    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-x2h67" podStartSLOduration=3.824840006 podStartE2EDuration="22.845301133s" totalImagesPullingTime="19.020461127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:45.845196987 +0000 UTC m=+23.854408855" watchObservedRunningTime="2026-07-31 14:48:45.845301133 +0000 UTC m=+23.854513001"
Jul 31 14:48:46.487131 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:46.487092    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:46.487328 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:46.487220    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:46.487635 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:46.487495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:46.487635 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:46.487591    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:46.721497 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:46.721256    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fbgpj" event={"ID":"8f19f238-718d-44d6-86af-799a206bc26e","Type":"ContainerStarted","Data":"cf8f2704956db958611bcc8c63e9a13316c134858d2467cfc9a1b965eaf9a1a4"}
Jul 31 14:48:46.739608 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:46.739510    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fbgpj" podStartSLOduration=4.7157716149999995 podStartE2EDuration="23.739493287s" totalImagesPullingTime="19.023721672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:46.739421648 +0000 UTC m=+24.748633518" watchObservedRunningTime="2026-07-31 14:48:46.739493287 +0000 UTC m=+24.748705156"
Jul 31 14:48:47.179408 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:47.179378    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 14:48:47.438335 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:47.438161    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T14:48:47.179404051Z","UUID":"4568e2df-9c17-40c7-a814-0b44b51ea193","Handler":null,"Name":"","Endpoint":""}
Jul 31 14:48:47.440089 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:47.440030    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 14:48:47.440089 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:47.440083    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 14:48:47.485036 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:47.485004    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:47.485234 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:47.485152    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:47.728248 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:47.728160    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk" event={"ID":"c22c54bd-d71b-4169-8fc7-1810529a2e67","Type":"ContainerStarted","Data":"9d21df94005e79c8989aab57b9e086b8c4a058ec5139355db887f2670635f528"}
Jul 31 14:48:47.730851 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:47.730821    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk" event={"ID":"fce1aee2-b141-4232-8fc0-d34fb333f976","Type":"ContainerStarted","Data":"28af1f21a17a415e0c06130ac70f627400665a21dc1211173136466446f43a31"}
Jul 31 14:48:48.485010 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:48.484979    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:48.485335 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:48.484985    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:48.485335 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:48.485074    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:48.485335 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:48.485162    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:49.484715 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:49.484670    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:49.485287 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:49.484811    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:50.485065 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:50.484869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:50.485912 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:50.484875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:50.485912 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:50.485149    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:50.485912 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:50.485219    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:50.575407 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:50.575368    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f4ptw"
Jul 31 14:48:50.576001 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:50.575980    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f4ptw"
Jul 31 14:48:50.741512 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:50.741427    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk" event={"ID":"c22c54bd-d71b-4169-8fc7-1810529a2e67","Type":"ContainerStarted","Data":"9ac5c86ef7d142a2b818a50a1cfc0cb23875b7934625fa015189d17124f26cde"}
Jul 31 14:48:50.741761 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:50.741723    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:50.743499 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:50.743465    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk" event={"ID":"fce1aee2-b141-4232-8fc0-d34fb333f976","Type":"ContainerStarted","Data":"c4f18e3bdc6289d1b1d91e0813be77b9a91a92f2d56669d1aa24e288d929ded7"}
Jul 31 14:48:50.745321 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:50.745300    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wc8q" event={"ID":"d391192c-cdbd-46ce-9098-444df02b6f1b","Type":"ContainerDied","Data":"4c90f6a51774865d8eb87bb7dc414fc85c9b8b22a557b286d9817afe6a6cce0d"}
Jul 31 14:48:50.745438 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:50.745293    2578 generic.go:356] "Generic (PLEG): container finished" podID="d391192c-cdbd-46ce-9098-444df02b6f1b" containerID="4c90f6a51774865d8eb87bb7dc414fc85c9b8b22a557b286d9817afe6a6cce0d" exitCode=0
Jul 31 14:48:50.745717 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:50.745704    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f4ptw"
Jul 31 14:48:50.746257 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:50.746237    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f4ptw"
Jul 31 14:48:50.758834 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:50.758816    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:50.806966 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:50.806919    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk" podStartSLOduration=8.721528403 podStartE2EDuration="27.80690391s" totalImagesPullingTime="19.085375507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:50.775792391 +0000 UTC m=+28.785004260" watchObservedRunningTime="2026-07-31 14:48:50.80690391 +0000 UTC m=+28.816115779"
Jul 31 14:48:50.826628 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:50.826557    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xghkk" podStartSLOduration=4.009653118 podStartE2EDuration="27.826546265s" totalImagesPullingTime="23.816893147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:50.826373889 +0000 UTC m=+28.835585758" watchObservedRunningTime="2026-07-31 14:48:50.826546265 +0000 UTC m=+28.835758133"
Jul 31 14:48:51.485238 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:51.485005    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:51.485712 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:51.485400    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:51.747693 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:51.747673    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 31 14:48:51.748255 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:51.748234    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:51.764958 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:51.764934    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:51.936301 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:51.936267    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kg6bm"]
Jul 31 14:48:51.936525 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:51.936424    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:51.936577 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:51.936538    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:51.936842 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:51.936823    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-97p2g"]
Jul 31 14:48:51.936912 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:51.936902    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:51.936997 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:51.936981    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:51.948981 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:51.948956    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xs6tw"]
Jul 31 14:48:51.949089 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:51.949051    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:51.949149 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:51.949124    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:52.592499 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:52.592465    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:48:52.750729 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:52.750701    2578 generic.go:356] "Generic (PLEG): container finished" podID="d391192c-cdbd-46ce-9098-444df02b6f1b" containerID="6c8b0ff9f3452f72c0aba6629db5429ec2df1c6e6562aece19d1610a1fd1b9c0" exitCode=0
Jul 31 14:48:52.750884 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:52.750775    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wc8q" event={"ID":"d391192c-cdbd-46ce-9098-444df02b6f1b","Type":"ContainerDied","Data":"6c8b0ff9f3452f72c0aba6629db5429ec2df1c6e6562aece19d1610a1fd1b9c0"}
Jul 31 14:48:53.484321 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:53.484289    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:53.484490 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:53.484407    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:53.484490 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:53.484422    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:53.484490 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:53.484435    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:53.484589 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:53.484493    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:53.484589 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:53.484577    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:53.755257 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:53.755170    2578 generic.go:356] "Generic (PLEG): container finished" podID="d391192c-cdbd-46ce-9098-444df02b6f1b" containerID="1d5a4ea8db80a3481e3a50b3c4a13da7731e630cdb76e07d188796638254bf70" exitCode=0
Jul 31 14:48:53.755721 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:53.755253    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wc8q" event={"ID":"d391192c-cdbd-46ce-9098-444df02b6f1b","Type":"ContainerDied","Data":"1d5a4ea8db80a3481e3a50b3c4a13da7731e630cdb76e07d188796638254bf70"}
Jul 31 14:48:55.485003 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:55.484806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:55.485475 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:55.484806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:55.485475 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:55.485113    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:55.485475 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:55.485177    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:55.485475 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:55.484806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:55.485475 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:55.485256    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:56.223815 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:56.223781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs\") pod \"network-metrics-daemon-97p2g\" (UID: \"352aa5e1-673c-4ede-a05a-80da4d81d2f9\") " pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:56.224011 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:56.223837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret\") pod \"global-pull-secret-syncer-kg6bm\" (UID: \"7fae56cb-8b98-4d8b-a508-62dac7aa73d1\") " pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:56.224011 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:56.223959    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:56.224143 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:56.224035    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs podName:352aa5e1-673c-4ede-a05a-80da4d81d2f9 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:28.224012885 +0000 UTC m=+66.233224754 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs") pod "network-metrics-daemon-97p2g" (UID: "352aa5e1-673c-4ede-a05a-80da4d81d2f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:56.224143 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:56.223965    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:56.224303 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:56.224129    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret podName:7fae56cb-8b98-4d8b-a508-62dac7aa73d1 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:28.224109295 +0000 UTC m=+66.233321162 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret") pod "global-pull-secret-syncer-kg6bm" (UID: "7fae56cb-8b98-4d8b-a508-62dac7aa73d1") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:56.324389 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:56.324301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvp6q\" (UniqueName: \"kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q\") pod \"network-check-target-xs6tw\" (UID: \"4138e89c-a29b-4e92-8c6e-99b232125ef8\") " pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:56.324546 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:56.324495    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:56.324546 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:56.324522    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:56.324546 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:56.324536    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pvp6q for pod openshift-network-diagnostics/network-check-target-xs6tw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:56.324687 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:56.324601    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q podName:4138e89c-a29b-4e92-8c6e-99b232125ef8 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:28.324582411 +0000 UTC m=+66.333794266 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pvp6q" (UniqueName: "kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q") pod "network-check-target-xs6tw" (UID: "4138e89c-a29b-4e92-8c6e-99b232125ef8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:57.484500 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:57.484463    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:57.484987 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:57.484463    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:57.484987 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:57.484600    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xs6tw" podUID="4138e89c-a29b-4e92-8c6e-99b232125ef8"
Jul 31 14:48:57.484987 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:57.484688    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg6bm" podUID="7fae56cb-8b98-4d8b-a508-62dac7aa73d1"
Jul 31 14:48:57.484987 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:57.484464    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:57.484987 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:57.484787    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97p2g" podUID="352aa5e1-673c-4ede-a05a-80da4d81d2f9"
Jul 31 14:48:57.768038 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:57.767992    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wc8q" event={"ID":"d391192c-cdbd-46ce-9098-444df02b6f1b","Type":"ContainerStarted","Data":"62326770fdd631799c2d6bc3b492f657fb0986538e8fb7f0556d79f8ef81c549"}
Jul 31 14:48:58.336901 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.336877    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeReady"
Jul 31 14:48:58.337030 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.337019    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 14:48:58.418589 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.418556    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bkzwx"]
Jul 31 14:48:58.420253 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.420237    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bkzwx"
Jul 31 14:48:58.422267 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.422246    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rcwl9"]
Jul 31 14:48:58.422694 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.422676    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 14:48:58.423200 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.423179    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-x9kr2\""
Jul 31 14:48:58.423278 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.423222    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 14:48:58.423278 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.423221    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 14:48:58.423946 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.423930    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:48:58.426293 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.426277    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 14:48:58.426399 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.426325    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 14:48:58.426608 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.426593    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p6mpf\""
Jul 31 14:48:58.435784 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.435765    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bkzwx"]
Jul 31 14:48:58.444405 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.444384    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rcwl9"]
Jul 31 14:48:58.452698 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.452679    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2wszh"]
Jul 31 14:48:58.454657 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.454645    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:48:58.457359 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.457332    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 14:48:58.457580 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.457567    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v2m8k\""
Jul 31 14:48:58.457867 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.457852    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 14:48:58.458292 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.458277    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 14:48:58.458554 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.458539    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 14:48:58.476939 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.476913    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2wszh"]
Jul 31 14:48:58.539917 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.539847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-config-volume\") pod \"dns-default-rcwl9\" (UID: \"4aa6bb07-f6ad-4a76-88e1-35fc595b4986\") " pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:48:58.539917 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.539888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls\") pod \"dns-default-rcwl9\" (UID: \"4aa6bb07-f6ad-4a76-88e1-35fc595b4986\") " pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:48:58.539917 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.539906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls25g\" (UniqueName: \"kubernetes.io/projected/964d007a-611d-4215-b4ba-8c4da12c2fbb-kube-api-access-ls25g\") pod \"ingress-canary-bkzwx\" (UID: \"964d007a-611d-4215-b4ba-8c4da12c2fbb\") " pod="openshift-ingress-canary/ingress-canary-bkzwx"
Jul 31 14:48:58.539917 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.539923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:48:58.540462 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.539941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-tmp-dir\") pod \"dns-default-rcwl9\" (UID: \"4aa6bb07-f6ad-4a76-88e1-35fc595b4986\") " pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:48:58.540462 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.539955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7b0e200-3697-4441-8b42-7333685b3cd4-crio-socket\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:48:58.540462 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.539969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whql2\" (UniqueName: \"kubernetes.io/projected/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-api-access-whql2\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:48:58.540462 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.539994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhd6p\" (UniqueName: \"kubernetes.io/projected/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-kube-api-access-qhd6p\") pod \"dns-default-rcwl9\" (UID: \"4aa6bb07-f6ad-4a76-88e1-35fc595b4986\") " pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:48:58.540462 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.540071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/964d007a-611d-4215-b4ba-8c4da12c2fbb-cert\") pod \"ingress-canary-bkzwx\" (UID: \"964d007a-611d-4215-b4ba-8c4da12c2fbb\") " pod="openshift-ingress-canary/ingress-canary-bkzwx"
Jul 31 14:48:58.540462 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.540095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:48:58.540462 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.540116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7b0e200-3697-4441-8b42-7333685b3cd4-data-volume\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:48:58.640706 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.640672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/964d007a-611d-4215-b4ba-8c4da12c2fbb-cert\") pod \"ingress-canary-bkzwx\" (UID: \"964d007a-611d-4215-b4ba-8c4da12c2fbb\") " pod="openshift-ingress-canary/ingress-canary-bkzwx"
Jul 31 14:48:58.640706 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.640711    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:48:58.640925 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.640732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7b0e200-3697-4441-8b42-7333685b3cd4-data-volume\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:48:58.640925 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.640765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-config-volume\") pod \"dns-default-rcwl9\" (UID: \"4aa6bb07-f6ad-4a76-88e1-35fc595b4986\") " pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:48:58.640925 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.640794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls\") pod \"dns-default-rcwl9\" (UID: \"4aa6bb07-f6ad-4a76-88e1-35fc595b4986\") " pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:48:58.640925 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:58.640806    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:48:58.640925 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.640813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls25g\" (UniqueName: \"kubernetes.io/projected/964d007a-611d-4215-b4ba-8c4da12c2fbb-kube-api-access-ls25g\") pod \"ingress-canary-bkzwx\" (UID: \"964d007a-611d-4215-b4ba-8c4da12c2fbb\") " pod="openshift-ingress-canary/ingress-canary-bkzwx"
Jul 31 14:48:58.640925 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:58.640891    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm podName:e7b0e200-3697-4441-8b42-7333685b3cd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:59.1408688 +0000 UTC m=+37.150080649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2wszh" (UID: "e7b0e200-3697-4441-8b42-7333685b3cd4") : configmap "kube-rbac-proxy" not found
Jul 31 14:48:58.641202 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.640931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:48:58.641202 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.640970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-tmp-dir\") pod \"dns-default-rcwl9\" (UID: \"4aa6bb07-f6ad-4a76-88e1-35fc595b4986\") " pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:48:58.641202 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.640994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7b0e200-3697-4441-8b42-7333685b3cd4-crio-socket\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:48:58.641202 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:58.640943    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:48:58.641202 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:58.641048    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:58.641202 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:58.641057    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls podName:4aa6bb07-f6ad-4a76-88e1-35fc595b4986 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:59.141045307 +0000 UTC m=+37.150257157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls") pod "dns-default-rcwl9" (UID: "4aa6bb07-f6ad-4a76-88e1-35fc595b4986") : secret "dns-default-metrics-tls" not found
Jul 31 14:48:58.641202 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:58.641137    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls podName:e7b0e200-3697-4441-8b42-7333685b3cd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:59.141117251 +0000 UTC m=+37.150329113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2wszh" (UID: "e7b0e200-3697-4441-8b42-7333685b3cd4") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:58.641202 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.641160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7b0e200-3697-4441-8b42-7333685b3cd4-crio-socket\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:48:58.641202 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.641165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whql2\" (UniqueName: \"kubernetes.io/projected/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-api-access-whql2\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:48:58.641202 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.641171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7b0e200-3697-4441-8b42-7333685b3cd4-data-volume\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:48:58.641607 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.641234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhd6p\" (UniqueName: \"kubernetes.io/projected/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-kube-api-access-qhd6p\") pod \"dns-default-rcwl9\" (UID: \"4aa6bb07-f6ad-4a76-88e1-35fc595b4986\") " pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:48:58.641607 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.641269    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-tmp-dir\") pod \"dns-default-rcwl9\" (UID: \"4aa6bb07-f6ad-4a76-88e1-35fc595b4986\") " pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:48:58.641607 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.641434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-config-volume\") pod \"dns-default-rcwl9\" (UID: \"4aa6bb07-f6ad-4a76-88e1-35fc595b4986\") " pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:48:58.644915 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.644897    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/964d007a-611d-4215-b4ba-8c4da12c2fbb-cert\") pod \"ingress-canary-bkzwx\" (UID: \"964d007a-611d-4215-b4ba-8c4da12c2fbb\") " pod="openshift-ingress-canary/ingress-canary-bkzwx"
Jul 31 14:48:58.655138 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.655110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whql2\" (UniqueName: \"kubernetes.io/projected/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-api-access-whql2\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:48:58.657607 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.657585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls25g\" (UniqueName: \"kubernetes.io/projected/964d007a-611d-4215-b4ba-8c4da12c2fbb-kube-api-access-ls25g\") pod \"ingress-canary-bkzwx\" (UID: \"964d007a-611d-4215-b4ba-8c4da12c2fbb\") " pod="openshift-ingress-canary/ingress-canary-bkzwx"
Jul 31 14:48:58.659241 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.659220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhd6p\" (UniqueName: \"kubernetes.io/projected/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-kube-api-access-qhd6p\") pod \"dns-default-rcwl9\" (UID: \"4aa6bb07-f6ad-4a76-88e1-35fc595b4986\") " pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:48:58.731151 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.731114    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bkzwx"
Jul 31 14:48:58.773585 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.773549    2578 generic.go:356] "Generic (PLEG): container finished" podID="d391192c-cdbd-46ce-9098-444df02b6f1b" containerID="62326770fdd631799c2d6bc3b492f657fb0986538e8fb7f0556d79f8ef81c549" exitCode=0
Jul 31 14:48:58.773770 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.773596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wc8q" event={"ID":"d391192c-cdbd-46ce-9098-444df02b6f1b","Type":"ContainerDied","Data":"62326770fdd631799c2d6bc3b492f657fb0986538e8fb7f0556d79f8ef81c549"}
Jul 31 14:48:58.875915 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:58.875705    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bkzwx"]
Jul 31 14:48:58.879557 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:48:58.879512    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964d007a_611d_4215_b4ba_8c4da12c2fbb.slice/crio-95bf4a0e13e7e238d010a98b59153b87a16e4bb60b8026c259a386cb086ca8a7 WatchSource:0}: Error finding container 95bf4a0e13e7e238d010a98b59153b87a16e4bb60b8026c259a386cb086ca8a7: Status 404 returned error can't find the container with id 95bf4a0e13e7e238d010a98b59153b87a16e4bb60b8026c259a386cb086ca8a7
Jul 31 14:48:59.145131 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:59.145032    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:48:59.145131 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:59.145094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls\") pod \"dns-default-rcwl9\" (UID: \"4aa6bb07-f6ad-4a76-88e1-35fc595b4986\") " pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:48:59.145131 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:59.145122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:48:59.145334 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:59.145159    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:48:59.145334 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:59.145227    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:59.145334 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:59.145241    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:48:59.145334 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:59.145268    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm podName:e7b0e200-3697-4441-8b42-7333685b3cd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:00.145252234 +0000 UTC m=+38.154464081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2wszh" (UID: "e7b0e200-3697-4441-8b42-7333685b3cd4") : configmap "kube-rbac-proxy" not found
Jul 31 14:48:59.145334 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:59.145287    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls podName:e7b0e200-3697-4441-8b42-7333685b3cd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:00.145277941 +0000 UTC m=+38.154489787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2wszh" (UID: "e7b0e200-3697-4441-8b42-7333685b3cd4") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:59.145334 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:48:59.145301    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls podName:4aa6bb07-f6ad-4a76-88e1-35fc595b4986 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:00.145293441 +0000 UTC m=+38.154505292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls") pod "dns-default-rcwl9" (UID: "4aa6bb07-f6ad-4a76-88e1-35fc595b4986") : secret "dns-default-metrics-tls" not found
Jul 31 14:48:59.177017 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:59.176994    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x2h67_be585c19-1ea7-4e53-8142-da3d9b2bc15a/dns-node-resolver/0.log"
Jul 31 14:48:59.484690 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:59.484611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:48:59.484852 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:59.484724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:48:59.484852 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:59.484761    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:48:59.487562 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:59.487542    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 14:48:59.487821 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:59.487803    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2kwb2\""
Jul 31 14:48:59.488224 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:59.488208    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 14:48:59.488943 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:59.488772    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 14:48:59.488943 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:59.488818    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qnc55\""
Jul 31 14:48:59.488943 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:59.488827    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 14:48:59.779769 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:59.779615    2578 generic.go:356] "Generic (PLEG): container finished" podID="d391192c-cdbd-46ce-9098-444df02b6f1b" containerID="f1a26f5c058dfd00c1b8647ebfa7ad006d96fd8e9445f38e24ad7a5af88c2003" exitCode=0
Jul 31 14:48:59.779769 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:59.779652    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wc8q" event={"ID":"d391192c-cdbd-46ce-9098-444df02b6f1b","Type":"ContainerDied","Data":"f1a26f5c058dfd00c1b8647ebfa7ad006d96fd8e9445f38e24ad7a5af88c2003"}
Jul 31 14:48:59.781006 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:59.780971    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bkzwx" event={"ID":"964d007a-611d-4215-b4ba-8c4da12c2fbb","Type":"ContainerStarted","Data":"95bf4a0e13e7e238d010a98b59153b87a16e4bb60b8026c259a386cb086ca8a7"}
Jul 31 14:48:59.939217 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:48:59.939189    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f74pb_82144e0f-aab6-4481-ab7f-86a85e384b20/node-ca/0.log"
Jul 31 14:49:00.154660 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:00.154574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:49:00.154660 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:00.154638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls\") pod \"dns-default-rcwl9\" (UID: \"4aa6bb07-f6ad-4a76-88e1-35fc595b4986\") " pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:49:00.154660 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:00.154657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:49:00.154941 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:00.154695    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:49:00.154941 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:00.154761    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:49:00.154941 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:00.154762    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm podName:e7b0e200-3697-4441-8b42-7333685b3cd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:02.15474338 +0000 UTC m=+40.163955231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2wszh" (UID: "e7b0e200-3697-4441-8b42-7333685b3cd4") : configmap "kube-rbac-proxy" not found
Jul 31 14:49:00.154941 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:00.154761    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:00.154941 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:00.154824    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls podName:4aa6bb07-f6ad-4a76-88e1-35fc595b4986 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:02.154809594 +0000 UTC m=+40.164021444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls") pod "dns-default-rcwl9" (UID: "4aa6bb07-f6ad-4a76-88e1-35fc595b4986") : secret "dns-default-metrics-tls" not found
Jul 31 14:49:00.154941 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:00.154869    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls podName:e7b0e200-3697-4441-8b42-7333685b3cd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:02.154855025 +0000 UTC m=+40.164066891 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2wszh" (UID: "e7b0e200-3697-4441-8b42-7333685b3cd4") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:00.192104 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:00.192074    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-vmvsm"]
Jul 31 14:49:00.194604 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:00.194586    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-vmvsm"
Jul 31 14:49:00.198590 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:00.198371    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cjd66\""
Jul 31 14:49:00.198590 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:00.198420    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 31 14:49:00.198590 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:00.198486    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 31 14:49:00.216950 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:00.216911    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-vmvsm"]
Jul 31 14:49:00.356514 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:00.356470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxfnh\" (UniqueName: \"kubernetes.io/projected/1db937c9-6ed5-422f-abd3-95849fd5f033-kube-api-access-fxfnh\") pod \"migrator-68bc745d87-vmvsm\" (UID: \"1db937c9-6ed5-422f-abd3-95849fd5f033\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-vmvsm"
Jul 31 14:49:00.457606 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:00.457522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxfnh\" (UniqueName: \"kubernetes.io/projected/1db937c9-6ed5-422f-abd3-95849fd5f033-kube-api-access-fxfnh\") pod \"migrator-68bc745d87-vmvsm\" (UID: \"1db937c9-6ed5-422f-abd3-95849fd5f033\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-vmvsm"
Jul 31 14:49:00.470652 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:00.470619    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxfnh\" (UniqueName: \"kubernetes.io/projected/1db937c9-6ed5-422f-abd3-95849fd5f033-kube-api-access-fxfnh\") pod \"migrator-68bc745d87-vmvsm\" (UID: \"1db937c9-6ed5-422f-abd3-95849fd5f033\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-vmvsm"
Jul 31 14:49:00.505592 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:00.505565    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-vmvsm"
Jul 31 14:49:00.787608 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:00.787529    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wc8q" event={"ID":"d391192c-cdbd-46ce-9098-444df02b6f1b","Type":"ContainerStarted","Data":"3ef37995c89b90c6e423724b181f361b816c0fed232b4a98131fb0af7fa46637"}
Jul 31 14:49:00.816906 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:00.816842    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4wc8q" podStartSLOduration=9.008772176 podStartE2EDuration="37.816820623s" totalImagesPullingTime="28.808048447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:00.815275074 +0000 UTC m=+38.824486944" watchObservedRunningTime="2026-07-31 14:49:00.816820623 +0000 UTC m=+38.826032496"
Jul 31 14:49:00.986931 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:00.986761    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-vmvsm"]
Jul 31 14:49:01.130940 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:49:01.130868    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db937c9_6ed5_422f_abd3_95849fd5f033.slice/crio-21c3fb8ad7fd79aec20ece7ea4b07c8e0045582bcf0ad960b9003166f781f15b WatchSource:0}: Error finding container 21c3fb8ad7fd79aec20ece7ea4b07c8e0045582bcf0ad960b9003166f781f15b: Status 404 returned error can't find the container with id 21c3fb8ad7fd79aec20ece7ea4b07c8e0045582bcf0ad960b9003166f781f15b
Jul 31 14:49:01.793329 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:01.793230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bkzwx" event={"ID":"964d007a-611d-4215-b4ba-8c4da12c2fbb","Type":"ContainerStarted","Data":"745f9a9d8a81a92915f4233d3e1d42dd51b9542050d3183a29b4b5753370ca44"}
Jul 31 14:49:01.794493 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:01.794466    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-vmvsm" event={"ID":"1db937c9-6ed5-422f-abd3-95849fd5f033","Type":"ContainerStarted","Data":"21c3fb8ad7fd79aec20ece7ea4b07c8e0045582bcf0ad960b9003166f781f15b"}
Jul 31 14:49:01.816941 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:01.816890    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bkzwx" podStartSLOduration=1.539295669 podStartE2EDuration="3.816873937s" totalImagesPullingTime="2.277578268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:01.816835049 +0000 UTC m=+39.826046922" watchObservedRunningTime="2026-07-31 14:49:01.816873937 +0000 UTC m=+39.826085805"
Jul 31 14:49:02.170057 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:02.169965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls\") pod \"dns-default-rcwl9\" (UID: \"4aa6bb07-f6ad-4a76-88e1-35fc595b4986\") " pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:49:02.170057 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:02.170020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:49:02.170274 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:02.170089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:49:02.170274 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:02.170121    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:49:02.170274 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:02.170175    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:49:02.170274 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:02.170185    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls podName:4aa6bb07-f6ad-4a76-88e1-35fc595b4986 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:06.170166409 +0000 UTC m=+44.179378275 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls") pod "dns-default-rcwl9" (UID: "4aa6bb07-f6ad-4a76-88e1-35fc595b4986") : secret "dns-default-metrics-tls" not found
Jul 31 14:49:02.170274 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:02.170220    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm podName:e7b0e200-3697-4441-8b42-7333685b3cd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:06.170210111 +0000 UTC m=+44.179421958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2wszh" (UID: "e7b0e200-3697-4441-8b42-7333685b3cd4") : configmap "kube-rbac-proxy" not found
Jul 31 14:49:02.170274 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:02.170273    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:02.170595 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:02.170302    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls podName:e7b0e200-3697-4441-8b42-7333685b3cd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:06.170292219 +0000 UTC m=+44.179504066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2wszh" (UID: "e7b0e200-3697-4441-8b42-7333685b3cd4") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:03.487132 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.487100    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-6vds4"]
Jul 31 14:49:03.509451 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.509426    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-6vds4"]
Jul 31 14:49:03.509580 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.509513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-6vds4"
Jul 31 14:49:03.515544 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.515500    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 31 14:49:03.515544 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.515516    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 31 14:49:03.516692 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.516675    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 31 14:49:03.516786 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.516771    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pvn6h\""
Jul 31 14:49:03.516853 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.516777    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 31 14:49:03.681936 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.681897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dc2acb32-f4a1-40c5-8461-5f2d20963ae6-signing-cabundle\") pod \"service-ca-98c88fbb5-6vds4\" (UID: \"dc2acb32-f4a1-40c5-8461-5f2d20963ae6\") " pod="openshift-service-ca/service-ca-98c88fbb5-6vds4"
Jul 31 14:49:03.682108 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.681985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dc2acb32-f4a1-40c5-8461-5f2d20963ae6-signing-key\") pod \"service-ca-98c88fbb5-6vds4\" (UID: \"dc2acb32-f4a1-40c5-8461-5f2d20963ae6\") " pod="openshift-service-ca/service-ca-98c88fbb5-6vds4"
Jul 31 14:49:03.682108 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.682024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvvrx\" (UniqueName: \"kubernetes.io/projected/dc2acb32-f4a1-40c5-8461-5f2d20963ae6-kube-api-access-jvvrx\") pod \"service-ca-98c88fbb5-6vds4\" (UID: \"dc2acb32-f4a1-40c5-8461-5f2d20963ae6\") " pod="openshift-service-ca/service-ca-98c88fbb5-6vds4"
Jul 31 14:49:03.782674 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.782607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dc2acb32-f4a1-40c5-8461-5f2d20963ae6-signing-key\") pod \"service-ca-98c88fbb5-6vds4\" (UID: \"dc2acb32-f4a1-40c5-8461-5f2d20963ae6\") " pod="openshift-service-ca/service-ca-98c88fbb5-6vds4"
Jul 31 14:49:03.782674 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.782665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvvrx\" (UniqueName: \"kubernetes.io/projected/dc2acb32-f4a1-40c5-8461-5f2d20963ae6-kube-api-access-jvvrx\") pod \"service-ca-98c88fbb5-6vds4\" (UID: \"dc2acb32-f4a1-40c5-8461-5f2d20963ae6\") " pod="openshift-service-ca/service-ca-98c88fbb5-6vds4"
Jul 31 14:49:03.782789 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.782730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dc2acb32-f4a1-40c5-8461-5f2d20963ae6-signing-cabundle\") pod \"service-ca-98c88fbb5-6vds4\" (UID: \"dc2acb32-f4a1-40c5-8461-5f2d20963ae6\") " pod="openshift-service-ca/service-ca-98c88fbb5-6vds4"
Jul 31 14:49:03.783502 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.783474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dc2acb32-f4a1-40c5-8461-5f2d20963ae6-signing-cabundle\") pod \"service-ca-98c88fbb5-6vds4\" (UID: \"dc2acb32-f4a1-40c5-8461-5f2d20963ae6\") " pod="openshift-service-ca/service-ca-98c88fbb5-6vds4"
Jul 31 14:49:03.786035 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.786013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dc2acb32-f4a1-40c5-8461-5f2d20963ae6-signing-key\") pod \"service-ca-98c88fbb5-6vds4\" (UID: \"dc2acb32-f4a1-40c5-8461-5f2d20963ae6\") " pod="openshift-service-ca/service-ca-98c88fbb5-6vds4"
Jul 31 14:49:03.793724 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.793701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvvrx\" (UniqueName: \"kubernetes.io/projected/dc2acb32-f4a1-40c5-8461-5f2d20963ae6-kube-api-access-jvvrx\") pod \"service-ca-98c88fbb5-6vds4\" (UID: \"dc2acb32-f4a1-40c5-8461-5f2d20963ae6\") " pod="openshift-service-ca/service-ca-98c88fbb5-6vds4"
Jul 31 14:49:03.801147 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.801123    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-vmvsm" event={"ID":"1db937c9-6ed5-422f-abd3-95849fd5f033","Type":"ContainerStarted","Data":"dda0bd4ab393af19468c85fdc349b9607aaaff5de2ae318766a9aaad10e8e7d3"}
Jul 31 14:49:03.801280 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.801169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-vmvsm" event={"ID":"1db937c9-6ed5-422f-abd3-95849fd5f033","Type":"ContainerStarted","Data":"eca2e8c115aee353bbe7f71457b4a43bebfaa79d015f37d8b284b4e8dc69fac7"}
Jul 31 14:49:03.818391 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.818366    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-6vds4"
Jul 31 14:49:03.827488 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.827441    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-vmvsm" podStartSLOduration=2.247717139 podStartE2EDuration="3.827427516s" totalImagesPullingTime="1.579710377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:03.827120905 +0000 UTC m=+41.836332773" watchObservedRunningTime="2026-07-31 14:49:03.827427516 +0000 UTC m=+41.836639429"
Jul 31 14:49:03.991013 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:03.990977    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-6vds4"]
Jul 31 14:49:03.993782 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:49:03.993754    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2acb32_f4a1_40c5_8461_5f2d20963ae6.slice/crio-8f33f8c91736ecb8acf0fffb2a313d82e65d885347740a312482f1e503f9969f WatchSource:0}: Error finding container 8f33f8c91736ecb8acf0fffb2a313d82e65d885347740a312482f1e503f9969f: Status 404 returned error can't find the container with id 8f33f8c91736ecb8acf0fffb2a313d82e65d885347740a312482f1e503f9969f
Jul 31 14:49:04.805319 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:04.805278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-6vds4" event={"ID":"dc2acb32-f4a1-40c5-8461-5f2d20963ae6","Type":"ContainerStarted","Data":"8f33f8c91736ecb8acf0fffb2a313d82e65d885347740a312482f1e503f9969f"}
Jul 31 14:49:06.202651 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:06.202606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls\") pod \"dns-default-rcwl9\" (UID: \"4aa6bb07-f6ad-4a76-88e1-35fc595b4986\") " pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:49:06.202651 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:06.202652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:49:06.203098 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:06.202701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:49:06.203098 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:06.202776    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:49:06.203098 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:06.202827    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:06.203098 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:06.202847    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls podName:4aa6bb07-f6ad-4a76-88e1-35fc595b4986 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:14.202831223 +0000 UTC m=+52.212043070 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls") pod "dns-default-rcwl9" (UID: "4aa6bb07-f6ad-4a76-88e1-35fc595b4986") : secret "dns-default-metrics-tls" not found
Jul 31 14:49:06.203098 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:06.202785    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:49:06.203098 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:06.202873    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls podName:e7b0e200-3697-4441-8b42-7333685b3cd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:14.202859064 +0000 UTC m=+52.212070916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2wszh" (UID: "e7b0e200-3697-4441-8b42-7333685b3cd4") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:06.203098 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:06.202891    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm podName:e7b0e200-3697-4441-8b42-7333685b3cd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:14.202882812 +0000 UTC m=+52.212094659 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2wszh" (UID: "e7b0e200-3697-4441-8b42-7333685b3cd4") : configmap "kube-rbac-proxy" not found
Jul 31 14:49:06.811244 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:06.811208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-6vds4" event={"ID":"dc2acb32-f4a1-40c5-8461-5f2d20963ae6","Type":"ContainerStarted","Data":"ca12ab2fe6e432d6b9556b4e1bc8922d29ec4ff698ad18a6c4a1d9a9cf42550b"}
Jul 31 14:49:06.829042 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:06.828991    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-6vds4" podStartSLOduration=1.48376618 podStartE2EDuration="3.828976212s" totalImagesPullingTime="2.345210032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:06.828311467 +0000 UTC m=+44.837523335" watchObservedRunningTime="2026-07-31 14:49:06.828976212 +0000 UTC m=+44.838188081"
Jul 31 14:49:14.257931 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:14.257892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:49:14.258450 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:14.257963    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls\") pod \"dns-default-rcwl9\" (UID: \"4aa6bb07-f6ad-4a76-88e1-35fc595b4986\") " pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:49:14.258450 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:14.257982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:49:14.258450 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:14.258073    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:14.258450 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:49:14.258125    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls podName:e7b0e200-3697-4441-8b42-7333685b3cd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:30.258112334 +0000 UTC m=+68.267324181 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2wszh" (UID: "e7b0e200-3697-4441-8b42-7333685b3cd4") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:14.258599 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:14.258523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7b0e200-3697-4441-8b42-7333685b3cd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:49:14.260477 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:14.260455    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4aa6bb07-f6ad-4a76-88e1-35fc595b4986-metrics-tls\") pod \"dns-default-rcwl9\" (UID: \"4aa6bb07-f6ad-4a76-88e1-35fc595b4986\") " pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:49:14.338024 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:14.337992    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:49:14.492363 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:14.492317    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rcwl9"]
Jul 31 14:49:14.495224 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:49:14.495195    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa6bb07_f6ad_4a76_88e1_35fc595b4986.slice/crio-8496ada1eafbabf9a43b255aed0c8c4dad9c67c94b04884845d2ce4c496f6076 WatchSource:0}: Error finding container 8496ada1eafbabf9a43b255aed0c8c4dad9c67c94b04884845d2ce4c496f6076: Status 404 returned error can't find the container with id 8496ada1eafbabf9a43b255aed0c8c4dad9c67c94b04884845d2ce4c496f6076
Jul 31 14:49:14.831690 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:14.831656    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rcwl9" event={"ID":"4aa6bb07-f6ad-4a76-88e1-35fc595b4986","Type":"ContainerStarted","Data":"8496ada1eafbabf9a43b255aed0c8c4dad9c67c94b04884845d2ce4c496f6076"}
Jul 31 14:49:16.839994 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:16.839957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rcwl9" event={"ID":"4aa6bb07-f6ad-4a76-88e1-35fc595b4986","Type":"ContainerStarted","Data":"894baecfe28fdb32556e8a3e1fae20ac9341b9c76f9f7aed6f285092245a3747"}
Jul 31 14:49:16.840384 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:16.840012    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rcwl9" event={"ID":"4aa6bb07-f6ad-4a76-88e1-35fc595b4986","Type":"ContainerStarted","Data":"1da5ab59dc6f901ec726bd6859ab91ce9d29df20d00d2301fa757301787a1418"}
Jul 31 14:49:16.840384 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:16.840061    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:49:16.864236 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:16.864192    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rcwl9" podStartSLOduration=16.961286701 podStartE2EDuration="18.864179244s" totalImagesPullingTime="1.902892543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:16.862596294 +0000 UTC m=+54.871808164" watchObservedRunningTime="2026-07-31 14:49:16.864179244 +0000 UTC m=+54.873391112"
Jul 31 14:49:23.769697 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:23.769664    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lhnwk"
Jul 31 14:49:25.604441 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.604408    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67f696bbd4-78xt2"]
Jul 31 14:49:25.607020 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.607002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67f696bbd4-78xt2"
Jul 31 14:49:25.611112 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.611090    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 31 14:49:25.612182 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.612160    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 31 14:49:25.612278 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.612251    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 31 14:49:25.612422 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.612406    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 31 14:49:25.615487 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.615470    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5tr2x\""
Jul 31 14:49:25.626539 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.626516    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"]
Jul 31 14:49:25.628707 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.628689    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67f696bbd4-78xt2"]
Jul 31 14:49:25.628822 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.628774    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.631730 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.631709    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 31 14:49:25.631827 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.631732    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 31 14:49:25.631827 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.631786    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 31 14:49:25.632020 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.632004    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 31 14:49:25.642473 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.642448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86dc7825-43a5-4bd7-9c2f-86b1cafcb0a8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67f696bbd4-78xt2\" (UID: \"86dc7825-43a5-4bd7-9c2f-86b1cafcb0a8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67f696bbd4-78xt2"
Jul 31 14:49:25.642613 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.642518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwkr4\" (UniqueName: \"kubernetes.io/projected/86dc7825-43a5-4bd7-9c2f-86b1cafcb0a8-kube-api-access-nwkr4\") pod \"managed-serviceaccount-addon-agent-67f696bbd4-78xt2\" (UID: \"86dc7825-43a5-4bd7-9c2f-86b1cafcb0a8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67f696bbd4-78xt2"
Jul 31 14:49:25.650053 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.650032    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"]
Jul 31 14:49:25.743252 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.743217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-ca\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.743476 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.743259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwkr4\" (UniqueName: \"kubernetes.io/projected/86dc7825-43a5-4bd7-9c2f-86b1cafcb0a8-kube-api-access-nwkr4\") pod \"managed-serviceaccount-addon-agent-67f696bbd4-78xt2\" (UID: \"86dc7825-43a5-4bd7-9c2f-86b1cafcb0a8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67f696bbd4-78xt2"
Jul 31 14:49:25.743476 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.743328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.743476 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.743378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh9jc\" (UniqueName: \"kubernetes.io/projected/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-kube-api-access-kh9jc\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.743476 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.743421    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-hub\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.743476 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.743439    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.743694 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.743508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86dc7825-43a5-4bd7-9c2f-86b1cafcb0a8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67f696bbd4-78xt2\" (UID: \"86dc7825-43a5-4bd7-9c2f-86b1cafcb0a8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67f696bbd4-78xt2"
Jul 31 14:49:25.743694 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.743535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.745933 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.745913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86dc7825-43a5-4bd7-9c2f-86b1cafcb0a8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67f696bbd4-78xt2\" (UID: \"86dc7825-43a5-4bd7-9c2f-86b1cafcb0a8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67f696bbd4-78xt2"
Jul 31 14:49:25.757459 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.757436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwkr4\" (UniqueName: \"kubernetes.io/projected/86dc7825-43a5-4bd7-9c2f-86b1cafcb0a8-kube-api-access-nwkr4\") pod \"managed-serviceaccount-addon-agent-67f696bbd4-78xt2\" (UID: \"86dc7825-43a5-4bd7-9c2f-86b1cafcb0a8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67f696bbd4-78xt2"
Jul 31 14:49:25.844862 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.844824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.845052 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.844883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-ca\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.845052 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.844910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.845052 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.844926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh9jc\" (UniqueName: \"kubernetes.io/projected/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-kube-api-access-kh9jc\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.845052 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.844951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-hub\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.845052 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.844966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.845708 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.845680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.847373 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.847333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.847473 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.847452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-ca\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.847878 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.847857    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-hub\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.847918 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.847891    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.857822 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.857772    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh9jc\" (UniqueName: \"kubernetes.io/projected/3c5ddf4f-2ce2-427d-80e9-7622cb8b682e-kube-api-access-kh9jc\") pod \"cluster-proxy-proxy-agent-844c6d949c-7hwh8\" (UID: \"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:25.924584 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.924554    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67f696bbd4-78xt2"
Jul 31 14:49:25.936975 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:25.936945    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"
Jul 31 14:49:26.049910 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:26.049887    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67f696bbd4-78xt2"]
Jul 31 14:49:26.052690 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:49:26.052658    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86dc7825_43a5_4bd7_9c2f_86b1cafcb0a8.slice/crio-e42178ef2090f7a90414553f0431d516377514537effc9dec57d13d4266a59a8 WatchSource:0}: Error finding container e42178ef2090f7a90414553f0431d516377514537effc9dec57d13d4266a59a8: Status 404 returned error can't find the container with id e42178ef2090f7a90414553f0431d516377514537effc9dec57d13d4266a59a8
Jul 31 14:49:26.077499 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:49:26.077473    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c5ddf4f_2ce2_427d_80e9_7622cb8b682e.slice/crio-f0a4ef8b5a63bf0b723082b3a5a34ffc69e75e68dcde18f40d6c4d727cd9bc01 WatchSource:0}: Error finding container f0a4ef8b5a63bf0b723082b3a5a34ffc69e75e68dcde18f40d6c4d727cd9bc01: Status 404 returned error can't find the container with id f0a4ef8b5a63bf0b723082b3a5a34ffc69e75e68dcde18f40d6c4d727cd9bc01
Jul 31 14:49:26.079323 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:26.079293    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8"]
Jul 31 14:49:26.846696 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:26.846002    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rcwl9"
Jul 31 14:49:26.875018 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:26.874947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8" event={"ID":"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e","Type":"ContainerStarted","Data":"f0a4ef8b5a63bf0b723082b3a5a34ffc69e75e68dcde18f40d6c4d727cd9bc01"}
Jul 31 14:49:26.877203 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:26.877175    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67f696bbd4-78xt2" event={"ID":"86dc7825-43a5-4bd7-9c2f-86b1cafcb0a8","Type":"ContainerStarted","Data":"e42178ef2090f7a90414553f0431d516377514537effc9dec57d13d4266a59a8"}
Jul 31 14:49:28.267100 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:28.267007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret\") pod \"global-pull-secret-syncer-kg6bm\" (UID: \"7fae56cb-8b98-4d8b-a508-62dac7aa73d1\") " pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:49:28.267537 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:28.267128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs\") pod \"network-metrics-daemon-97p2g\" (UID: \"352aa5e1-673c-4ede-a05a-80da4d81d2f9\") " pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:49:28.270461 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:28.270434    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 14:49:28.270461 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:28.270448    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 14:49:28.280460 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:28.280430    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/352aa5e1-673c-4ede-a05a-80da4d81d2f9-metrics-certs\") pod \"network-metrics-daemon-97p2g\" (UID: \"352aa5e1-673c-4ede-a05a-80da4d81d2f9\") " pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:49:28.281270 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:28.281241    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fae56cb-8b98-4d8b-a508-62dac7aa73d1-original-pull-secret\") pod \"global-pull-secret-syncer-kg6bm\" (UID: \"7fae56cb-8b98-4d8b-a508-62dac7aa73d1\") " pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:49:28.295760 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:28.295739    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg6bm"
Jul 31 14:49:28.310938 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:28.310909    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qnc55\""
Jul 31 14:49:28.318603 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:28.318405    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97p2g"
Jul 31 14:49:28.368423 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:28.368395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvp6q\" (UniqueName: \"kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q\") pod \"network-check-target-xs6tw\" (UID: \"4138e89c-a29b-4e92-8c6e-99b232125ef8\") " pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:49:28.372426 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:28.372380    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 14:49:28.381735 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:28.381712    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 14:49:28.392490 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:28.392433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvp6q\" (UniqueName: \"kubernetes.io/projected/4138e89c-a29b-4e92-8c6e-99b232125ef8-kube-api-access-pvp6q\") pod \"network-check-target-xs6tw\" (UID: \"4138e89c-a29b-4e92-8c6e-99b232125ef8\") " pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:49:28.615712 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:28.615633    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2kwb2\""
Jul 31 14:49:28.623221 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:28.623190    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:49:30.282173 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:30.282136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:49:30.284702 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:30.284670    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7b0e200-3697-4441-8b42-7333685b3cd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2wszh\" (UID: \"e7b0e200-3697-4441-8b42-7333685b3cd4\") " pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:49:30.566733 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:30.566660    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v2m8k\""
Jul 31 14:49:30.573687 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:30.573660    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2wszh"
Jul 31 14:49:31.167753 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:31.167580    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kg6bm"]
Jul 31 14:49:31.173086 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:49:31.173056    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fae56cb_8b98_4d8b_a508_62dac7aa73d1.slice/crio-d4b3b08590bfd282a282731cb0ac6a5bbfb6234af9d00a5b31dcdb562b04f9d9 WatchSource:0}: Error finding container d4b3b08590bfd282a282731cb0ac6a5bbfb6234af9d00a5b31dcdb562b04f9d9: Status 404 returned error can't find the container with id d4b3b08590bfd282a282731cb0ac6a5bbfb6234af9d00a5b31dcdb562b04f9d9
Jul 31 14:49:31.186220 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:31.186192    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-97p2g"]
Jul 31 14:49:31.187240 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:49:31.187216    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352aa5e1_673c_4ede_a05a_80da4d81d2f9.slice/crio-88a86cdb59ea11a7b7a7c6e30aa94ad0a813fcf0758d62e8978e0732a00de87f WatchSource:0}: Error finding container 88a86cdb59ea11a7b7a7c6e30aa94ad0a813fcf0758d62e8978e0732a00de87f: Status 404 returned error can't find the container with id 88a86cdb59ea11a7b7a7c6e30aa94ad0a813fcf0758d62e8978e0732a00de87f
Jul 31 14:49:31.435138 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:31.435116    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xs6tw"]
Jul 31 14:49:31.437317 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:49:31.437291    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4138e89c_a29b_4e92_8c6e_99b232125ef8.slice/crio-3b581a8a5a8b65439732b1f9dbb04cf530bbff2adf667acee26d64553fdf995e WatchSource:0}: Error finding container 3b581a8a5a8b65439732b1f9dbb04cf530bbff2adf667acee26d64553fdf995e: Status 404 returned error can't find the container with id 3b581a8a5a8b65439732b1f9dbb04cf530bbff2adf667acee26d64553fdf995e
Jul 31 14:49:31.442786 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:31.442766    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2wszh"]
Jul 31 14:49:31.445129 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:49:31.445099    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b0e200_3697_4441_8b42_7333685b3cd4.slice/crio-91170e9fab742e289c878feb15dc696c487ab5fa3209f19f6e3b8f6ce405f428 WatchSource:0}: Error finding container 91170e9fab742e289c878feb15dc696c487ab5fa3209f19f6e3b8f6ce405f428: Status 404 returned error can't find the container with id 91170e9fab742e289c878feb15dc696c487ab5fa3209f19f6e3b8f6ce405f428
Jul 31 14:49:31.895097 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:31.895018    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xs6tw" event={"ID":"4138e89c-a29b-4e92-8c6e-99b232125ef8","Type":"ContainerStarted","Data":"3b581a8a5a8b65439732b1f9dbb04cf530bbff2adf667acee26d64553fdf995e"}
Jul 31 14:49:31.896419 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:31.896388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-97p2g" event={"ID":"352aa5e1-673c-4ede-a05a-80da4d81d2f9","Type":"ContainerStarted","Data":"88a86cdb59ea11a7b7a7c6e30aa94ad0a813fcf0758d62e8978e0732a00de87f"}
Jul 31 14:49:31.898700 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:31.898662    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8" event={"ID":"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e","Type":"ContainerStarted","Data":"d70921550ddc4be9b2536b17ef5f8862d2586d911df29dfe708bed991e241743"}
Jul 31 14:49:31.898806 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:31.898718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8" event={"ID":"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e","Type":"ContainerStarted","Data":"39c03a3c422d0571dfc4af9b2be7c9b07899ee2cd7d718e94c468b191f8483f8"}
Jul 31 14:49:31.898806 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:31.898740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8" event={"ID":"3c5ddf4f-2ce2-427d-80e9-7622cb8b682e","Type":"ContainerStarted","Data":"2acaf48ef9ea832246b86b0f19d9de7301fb29476dc90a4dae305360eb9f5390"}
Jul 31 14:49:31.900287 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:31.900265    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2wszh" event={"ID":"e7b0e200-3697-4441-8b42-7333685b3cd4","Type":"ContainerStarted","Data":"e51ecf8413332e4d4d8b2e35795342185ffe4737f8be09ff4095197f29e31365"}
Jul 31 14:49:31.900386 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:31.900312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2wszh" event={"ID":"e7b0e200-3697-4441-8b42-7333685b3cd4","Type":"ContainerStarted","Data":"91170e9fab742e289c878feb15dc696c487ab5fa3209f19f6e3b8f6ce405f428"}
Jul 31 14:49:31.901994 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:31.901949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67f696bbd4-78xt2" event={"ID":"86dc7825-43a5-4bd7-9c2f-86b1cafcb0a8","Type":"ContainerStarted","Data":"46f20b9013e61046b8ccab7e17b789956e0cf3a0698c1e3cfd24d17f2193a7b4"}
Jul 31 14:49:31.903189 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:31.903162    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kg6bm" event={"ID":"7fae56cb-8b98-4d8b-a508-62dac7aa73d1","Type":"ContainerStarted","Data":"d4b3b08590bfd282a282731cb0ac6a5bbfb6234af9d00a5b31dcdb562b04f9d9"}
Jul 31 14:49:31.922395 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:31.922325    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-844c6d949c-7hwh8" podStartSLOduration=1.9861532149999999 podStartE2EDuration="6.922312431s" totalImagesPullingTime="4.936159216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:31.920649447 +0000 UTC m=+69.929861328" watchObservedRunningTime="2026-07-31 14:49:31.922312431 +0000 UTC m=+69.931524300"
Jul 31 14:49:31.938337 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:31.937912    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67f696bbd4-78xt2" podStartSLOduration=1.970586076 podStartE2EDuration="6.937898736s" totalImagesPullingTime="4.96731266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:31.937602599 +0000 UTC m=+69.946814468" watchObservedRunningTime="2026-07-31 14:49:31.937898736 +0000 UTC m=+69.947110605"
Jul 31 14:49:33.915016 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:33.914971    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-97p2g" event={"ID":"352aa5e1-673c-4ede-a05a-80da4d81d2f9","Type":"ContainerStarted","Data":"f3d5216f6de0deeca8c99abfb4f78c7c73cb68fe238bd2cb4e8bcf7dda426a93"}
Jul 31 14:49:33.915522 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:33.915087    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-97p2g" event={"ID":"352aa5e1-673c-4ede-a05a-80da4d81d2f9","Type":"ContainerStarted","Data":"5575d9058977a70eefe11b3d034f778bbe08881670a1f6877c35c4325f8d3051"}
Jul 31 14:49:33.917435 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:33.917406    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2wszh" event={"ID":"e7b0e200-3697-4441-8b42-7333685b3cd4","Type":"ContainerStarted","Data":"d934a6b046a2c2ab9fd2c96bbb4bf816f9e5ef26f82a671d39fe138c38188004"}
Jul 31 14:49:33.937095 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:33.937034    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-97p2g" podStartSLOduration=69.298793241 podStartE2EDuration="1m10.937013479s" totalImagesPullingTime="1.638220238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:33.935782528 +0000 UTC m=+71.944994398" watchObservedRunningTime="2026-07-31 14:49:33.937013479 +0000 UTC m=+71.946225351"
Jul 31 14:49:36.929437 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:36.929399    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xs6tw" event={"ID":"4138e89c-a29b-4e92-8c6e-99b232125ef8","Type":"ContainerStarted","Data":"ba850573842a62b591d54478c038f6a49225b6f53fa87c4aef8b067d84e0923d"}
Jul 31 14:49:36.929904 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:36.929521    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:49:36.931530 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:36.931457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2wszh" event={"ID":"e7b0e200-3697-4441-8b42-7333685b3cd4","Type":"ContainerStarted","Data":"61b4e486bb190e61c46b4d0f7e26f5440e45ebfab6705997dad0c4211c2d0fcb"}
Jul 31 14:49:36.932932 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:36.932907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kg6bm" event={"ID":"7fae56cb-8b98-4d8b-a508-62dac7aa73d1","Type":"ContainerStarted","Data":"a05744b0ce1b6ffc669bbb1f838dedeb778745e11d39d0fdd394a9180072ce73"}
Jul 31 14:49:36.949152 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:36.949107    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xs6tw" podStartSLOduration=68.631320137 podStartE2EDuration="1m13.949094415s" totalImagesPullingTime="5.317774278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:36.947596396 +0000 UTC m=+74.956808268" watchObservedRunningTime="2026-07-31 14:49:36.949094415 +0000 UTC m=+74.958306283"
Jul 31 14:49:36.967068 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:36.967028    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kg6bm" podStartSLOduration=68.396868476 podStartE2EDuration="1m13.967014412s" totalImagesPullingTime="5.570145936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:36.966291945 +0000 UTC m=+74.975503815" watchObservedRunningTime="2026-07-31 14:49:36.967014412 +0000 UTC m=+74.976226282"
Jul 31 14:49:36.990256 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:36.990148    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2wszh" podStartSLOduration=33.878077568 podStartE2EDuration="38.990129634s" totalImagesPullingTime="5.112052066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:36.988226247 +0000 UTC m=+74.997438108" watchObservedRunningTime="2026-07-31 14:49:36.990129634 +0000 UTC m=+74.999341504"
Jul 31 14:49:41.256256 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.256223    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tgc6f"]
Jul 31 14:49:41.258413 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.258396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.261381 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.261343    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 14:49:41.261521 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.261500    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 14:49:41.261593 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.261571    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 14:49:41.261694 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.261677    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 14:49:41.262536 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.262513    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 14:49:41.262658 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.262541    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5lzwk\""
Jul 31 14:49:41.262658 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.262564    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 14:49:41.362551 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.362522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b4ffa460-2438-48b3-84b8-049537740158-node-exporter-accelerators-collector-config\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.362551 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.362555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b4ffa460-2438-48b3-84b8-049537740158-node-exporter-textfile\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.362815 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.362579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b4ffa460-2438-48b3-84b8-049537740158-node-exporter-wtmp\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.362815 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.362607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4ffa460-2438-48b3-84b8-049537740158-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.362815 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.362662    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b4ffa460-2438-48b3-84b8-049537740158-root\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.362815 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.362696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4ffa460-2438-48b3-84b8-049537740158-sys\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.362815 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.362726    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b4ffa460-2438-48b3-84b8-049537740158-node-exporter-tls\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.362815 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.362743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4ffa460-2438-48b3-84b8-049537740158-metrics-client-ca\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.362815 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.362770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jppdb\" (UniqueName: \"kubernetes.io/projected/b4ffa460-2438-48b3-84b8-049537740158-kube-api-access-jppdb\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.463742 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.463707    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4ffa460-2438-48b3-84b8-049537740158-sys\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.463937 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.463751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b4ffa460-2438-48b3-84b8-049537740158-node-exporter-tls\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.463937 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.463775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4ffa460-2438-48b3-84b8-049537740158-metrics-client-ca\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.463937 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.463822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4ffa460-2438-48b3-84b8-049537740158-sys\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.464098 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.463944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jppdb\" (UniqueName: \"kubernetes.io/projected/b4ffa460-2438-48b3-84b8-049537740158-kube-api-access-jppdb\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.464098 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.463997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b4ffa460-2438-48b3-84b8-049537740158-node-exporter-accelerators-collector-config\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.464098 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.464018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b4ffa460-2438-48b3-84b8-049537740158-node-exporter-textfile\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.464098 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.464045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b4ffa460-2438-48b3-84b8-049537740158-node-exporter-wtmp\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.464098 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.464082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4ffa460-2438-48b3-84b8-049537740158-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.464330 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.464121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b4ffa460-2438-48b3-84b8-049537740158-root\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.464330 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.464205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b4ffa460-2438-48b3-84b8-049537740158-root\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.464330 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.464316    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b4ffa460-2438-48b3-84b8-049537740158-node-exporter-textfile\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.464498 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.464337    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b4ffa460-2438-48b3-84b8-049537740158-node-exporter-wtmp\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.464498 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.464474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4ffa460-2438-48b3-84b8-049537740158-metrics-client-ca\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.464594 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.464498    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b4ffa460-2438-48b3-84b8-049537740158-node-exporter-accelerators-collector-config\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.466214 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.466196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b4ffa460-2438-48b3-84b8-049537740158-node-exporter-tls\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.466282 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.466260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4ffa460-2438-48b3-84b8-049537740158-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.474681 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.474659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jppdb\" (UniqueName: \"kubernetes.io/projected/b4ffa460-2438-48b3-84b8-049537740158-kube-api-access-jppdb\") pod \"node-exporter-tgc6f\" (UID: \"b4ffa460-2438-48b3-84b8-049537740158\") " pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.569671 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.569591    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tgc6f"
Jul 31 14:49:41.580428 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:49:41.580400    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ffa460_2438_48b3_84b8_049537740158.slice/crio-f45edfedea6f4f3eb9d003e3b833b28f48f0222bd3d9d99c79c7b55aa31fffb8 WatchSource:0}: Error finding container f45edfedea6f4f3eb9d003e3b833b28f48f0222bd3d9d99c79c7b55aa31fffb8: Status 404 returned error can't find the container with id f45edfedea6f4f3eb9d003e3b833b28f48f0222bd3d9d99c79c7b55aa31fffb8
Jul 31 14:49:41.949259 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:41.949172    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tgc6f" event={"ID":"b4ffa460-2438-48b3-84b8-049537740158","Type":"ContainerStarted","Data":"f45edfedea6f4f3eb9d003e3b833b28f48f0222bd3d9d99c79c7b55aa31fffb8"}
Jul 31 14:49:42.953558 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:42.953521    2578 generic.go:356] "Generic (PLEG): container finished" podID="b4ffa460-2438-48b3-84b8-049537740158" containerID="6155ec26bf7367d32937aa83560176027aada03389950f46758665e40b300f77" exitCode=0
Jul 31 14:49:42.953933 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:42.953588    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tgc6f" event={"ID":"b4ffa460-2438-48b3-84b8-049537740158","Type":"ContainerDied","Data":"6155ec26bf7367d32937aa83560176027aada03389950f46758665e40b300f77"}
Jul 31 14:49:43.958992 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:43.958957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tgc6f" event={"ID":"b4ffa460-2438-48b3-84b8-049537740158","Type":"ContainerStarted","Data":"b807a9534486916c913aeb2c305f9c438b097876db051be01306a1a82ac38d8f"}
Jul 31 14:49:43.959372 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:43.959011    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tgc6f" event={"ID":"b4ffa460-2438-48b3-84b8-049537740158","Type":"ContainerStarted","Data":"7fa06a242ac8ff2eed4c30d47ea52c69c8cf800dc9ea17aa5a1a0a2f588cb310"}
Jul 31 14:49:43.985755 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:49:43.985706    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tgc6f" podStartSLOduration=2.065011867 podStartE2EDuration="2.985692739s" totalImagesPullingTime="857.320565ms" totalInitContainerRuntime="63.360307ms" isStatefulPod=true podCreationTimestamp="2026-07-31 14:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:43.9836587 +0000 UTC m=+81.992870570" watchObservedRunningTime="2026-07-31 14:49:43.985692739 +0000 UTC m=+81.994904608"
Jul 31 14:50:07.938558 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:50:07.938523    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xs6tw"
Jul 31 14:53:22.370999 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:22.370972    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 14:53:53.727490 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:53.727453    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm"]
Jul 31 14:53:53.729600 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:53.729582    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm"
Jul 31 14:53:53.732317 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:53.732294    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 31 14:53:53.732442 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:53.732294    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 31 14:53:53.733506 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:53.733486    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 31 14:53:53.733600 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:53.733542    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 31 14:53:53.733600 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:53.733550    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-zpntd\""
Jul 31 14:53:53.733713 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:53.733604    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 31 14:53:53.739508 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:53.739483    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm"]
Jul 31 14:53:53.874467 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:53.874426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbg5h\" (UniqueName: \"kubernetes.io/projected/0f4236f7-3aca-4406-aff5-7c13b63fd879-kube-api-access-sbg5h\") pod \"keda-metrics-apiserver-55557df8c-qpkkm\" (UID: \"0f4236f7-3aca-4406-aff5-7c13b63fd879\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm"
Jul 31 14:53:53.874633 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:53.874478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/0f4236f7-3aca-4406-aff5-7c13b63fd879-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-qpkkm\" (UID: \"0f4236f7-3aca-4406-aff5-7c13b63fd879\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm"
Jul 31 14:53:53.874633 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:53.874525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0f4236f7-3aca-4406-aff5-7c13b63fd879-certificates\") pod \"keda-metrics-apiserver-55557df8c-qpkkm\" (UID: \"0f4236f7-3aca-4406-aff5-7c13b63fd879\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm"
Jul 31 14:53:53.975200 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:53.975168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/0f4236f7-3aca-4406-aff5-7c13b63fd879-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-qpkkm\" (UID: \"0f4236f7-3aca-4406-aff5-7c13b63fd879\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm"
Jul 31 14:53:53.975200 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:53.975204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0f4236f7-3aca-4406-aff5-7c13b63fd879-certificates\") pod \"keda-metrics-apiserver-55557df8c-qpkkm\" (UID: \"0f4236f7-3aca-4406-aff5-7c13b63fd879\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm"
Jul 31 14:53:53.975435 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:53.975267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbg5h\" (UniqueName: \"kubernetes.io/projected/0f4236f7-3aca-4406-aff5-7c13b63fd879-kube-api-access-sbg5h\") pod \"keda-metrics-apiserver-55557df8c-qpkkm\" (UID: \"0f4236f7-3aca-4406-aff5-7c13b63fd879\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm"
Jul 31 14:53:53.975435 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:53:53.975407    2578 secret.go:281] references non-existent secret key: tls.crt
Jul 31 14:53:53.975435 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:53:53.975426    2578 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 31 14:53:53.975541 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:53:53.975444    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm: references non-existent secret key: tls.crt
Jul 31 14:53:53.975541 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:53:53.975507    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f4236f7-3aca-4406-aff5-7c13b63fd879-certificates podName:0f4236f7-3aca-4406-aff5-7c13b63fd879 nodeName:}" failed. No retries permitted until 2026-07-31 14:53:54.475486715 +0000 UTC m=+332.484698578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/0f4236f7-3aca-4406-aff5-7c13b63fd879-certificates") pod "keda-metrics-apiserver-55557df8c-qpkkm" (UID: "0f4236f7-3aca-4406-aff5-7c13b63fd879") : references non-existent secret key: tls.crt
Jul 31 14:53:53.975667 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:53.975650    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/0f4236f7-3aca-4406-aff5-7c13b63fd879-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-qpkkm\" (UID: \"0f4236f7-3aca-4406-aff5-7c13b63fd879\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm"
Jul 31 14:53:53.984491 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:53.984443    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbg5h\" (UniqueName: \"kubernetes.io/projected/0f4236f7-3aca-4406-aff5-7c13b63fd879-kube-api-access-sbg5h\") pod \"keda-metrics-apiserver-55557df8c-qpkkm\" (UID: \"0f4236f7-3aca-4406-aff5-7c13b63fd879\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm"
Jul 31 14:53:54.480054 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:54.480020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0f4236f7-3aca-4406-aff5-7c13b63fd879-certificates\") pod \"keda-metrics-apiserver-55557df8c-qpkkm\" (UID: \"0f4236f7-3aca-4406-aff5-7c13b63fd879\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm"
Jul 31 14:53:54.482428 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:54.482410    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0f4236f7-3aca-4406-aff5-7c13b63fd879-certificates\") pod \"keda-metrics-apiserver-55557df8c-qpkkm\" (UID: \"0f4236f7-3aca-4406-aff5-7c13b63fd879\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm"
Jul 31 14:53:54.640288 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:54.640255    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm"
Jul 31 14:53:54.767373 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:54.767273    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm"]
Jul 31 14:53:54.770375 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:53:54.770337    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4236f7_3aca_4406_aff5_7c13b63fd879.slice/crio-84063ca4968eac470d399a186fe187075c54439914f95b5b7591f96a13ea72ec WatchSource:0}: Error finding container 84063ca4968eac470d399a186fe187075c54439914f95b5b7591f96a13ea72ec: Status 404 returned error can't find the container with id 84063ca4968eac470d399a186fe187075c54439914f95b5b7591f96a13ea72ec
Jul 31 14:53:54.772037 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:54.772020    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:53:55.748257 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:53:55.748206    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm" event={"ID":"0f4236f7-3aca-4406-aff5-7c13b63fd879","Type":"ContainerStarted","Data":"84063ca4968eac470d399a186fe187075c54439914f95b5b7591f96a13ea72ec"}
Jul 31 14:54:03.777153 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:54:03.777061    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm" event={"ID":"0f4236f7-3aca-4406-aff5-7c13b63fd879","Type":"ContainerStarted","Data":"2e37b939a813a7d880ec24eaf90586be441883dc0b264884f4fd621c6b791a65"}
Jul 31 14:54:03.777609 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:54:03.777177    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm"
Jul 31 14:54:03.797611 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:54:03.797556    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm" podStartSLOduration=2.748022268 podStartE2EDuration="10.797540822s" totalImagesPullingTime="8.049518554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:54:03.795468106 +0000 UTC m=+341.804679978" watchObservedRunningTime="2026-07-31 14:54:03.797540822 +0000 UTC m=+341.806752691"
Jul 31 14:54:14.785764 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:54:14.785687    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-qpkkm"
Jul 31 14:55:23.487920 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:23.487884    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6bzw"]
Jul 31 14:55:23.490861 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:23.490845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6bzw"
Jul 31 14:55:23.493549 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:23.493527    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:55:23.493673 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:23.493544    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-zpdcj\""
Jul 31 14:55:23.494864 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:23.494849    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 31 14:55:23.499495 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:23.499474    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6bzw"]
Jul 31 14:55:23.585423 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:23.585394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75538cda-d1dc-470a-9322-fbbc951ad737-tmp\") pod \"openshift-lws-operator-fd8ccff4c-g6bzw\" (UID: \"75538cda-d1dc-470a-9322-fbbc951ad737\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6bzw"
Jul 31 14:55:23.585578 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:23.585447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j6pk\" (UniqueName: \"kubernetes.io/projected/75538cda-d1dc-470a-9322-fbbc951ad737-kube-api-access-4j6pk\") pod \"openshift-lws-operator-fd8ccff4c-g6bzw\" (UID: \"75538cda-d1dc-470a-9322-fbbc951ad737\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6bzw"
Jul 31 14:55:23.685989 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:23.685942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j6pk\" (UniqueName: \"kubernetes.io/projected/75538cda-d1dc-470a-9322-fbbc951ad737-kube-api-access-4j6pk\") pod \"openshift-lws-operator-fd8ccff4c-g6bzw\" (UID: \"75538cda-d1dc-470a-9322-fbbc951ad737\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6bzw"
Jul 31 14:55:23.685989 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:23.685999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75538cda-d1dc-470a-9322-fbbc951ad737-tmp\") pod \"openshift-lws-operator-fd8ccff4c-g6bzw\" (UID: \"75538cda-d1dc-470a-9322-fbbc951ad737\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6bzw"
Jul 31 14:55:23.686338 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:23.686323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75538cda-d1dc-470a-9322-fbbc951ad737-tmp\") pod \"openshift-lws-operator-fd8ccff4c-g6bzw\" (UID: \"75538cda-d1dc-470a-9322-fbbc951ad737\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6bzw"
Jul 31 14:55:23.695773 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:23.695750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j6pk\" (UniqueName: \"kubernetes.io/projected/75538cda-d1dc-470a-9322-fbbc951ad737-kube-api-access-4j6pk\") pod \"openshift-lws-operator-fd8ccff4c-g6bzw\" (UID: \"75538cda-d1dc-470a-9322-fbbc951ad737\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6bzw"
Jul 31 14:55:23.800527 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:23.800450    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6bzw"
Jul 31 14:55:23.922976 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:23.922942    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6bzw"]
Jul 31 14:55:23.926122 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:55:23.926090    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75538cda_d1dc_470a_9322_fbbc951ad737.slice/crio-dd73a91b134bd030a2bc8d8d7a7c931aaac70c893a7cca759d181ef651432302 WatchSource:0}: Error finding container dd73a91b134bd030a2bc8d8d7a7c931aaac70c893a7cca759d181ef651432302: Status 404 returned error can't find the container with id dd73a91b134bd030a2bc8d8d7a7c931aaac70c893a7cca759d181ef651432302
Jul 31 14:55:24.027849 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:24.027811    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6bzw" event={"ID":"75538cda-d1dc-470a-9322-fbbc951ad737","Type":"ContainerStarted","Data":"dd73a91b134bd030a2bc8d8d7a7c931aaac70c893a7cca759d181ef651432302"}
Jul 31 14:55:27.040799 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:27.040765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6bzw" event={"ID":"75538cda-d1dc-470a-9322-fbbc951ad737","Type":"ContainerStarted","Data":"ff40152127a9b1eea5149064eb74fbd618ebb46ab901c6ac2e51eebfaf1a0ef7"}
Jul 31 14:55:27.058290 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:27.058240    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g6bzw" podStartSLOduration=1.408442129 podStartE2EDuration="4.058225673s" totalImagesPullingTime="2.649783544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:55:27.057558723 +0000 UTC m=+425.066770593" watchObservedRunningTime="2026-07-31 14:55:27.058225673 +0000 UTC m=+425.067437541"
Jul 31 14:55:46.370819 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.370748    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"]
Jul 31 14:55:46.373782 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.373766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.378503 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.378482    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 31 14:55:46.379540 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.379517    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-kvrbh\""
Jul 31 14:55:46.379638 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.379517    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 31 14:55:46.380471 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.380084    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 31 14:55:46.409911 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.409879    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"]
Jul 31 14:55:46.446216 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.446186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9020da08-566c-41f1-bf8b-339a5d8ed640-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.446377 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.446219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9020da08-566c-41f1-bf8b-339a5d8ed640-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.446377 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.446244    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9020da08-566c-41f1-bf8b-339a5d8ed640-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.446377 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.446286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9020da08-566c-41f1-bf8b-339a5d8ed640-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.446377 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.446333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9020da08-566c-41f1-bf8b-339a5d8ed640-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.446377 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.446372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxxvd\" (UniqueName: \"kubernetes.io/projected/9020da08-566c-41f1-bf8b-339a5d8ed640-kube-api-access-bxxvd\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.446548 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.446419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9020da08-566c-41f1-bf8b-339a5d8ed640-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.446548 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.446466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9020da08-566c-41f1-bf8b-339a5d8ed640-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.446548 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.446481    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9020da08-566c-41f1-bf8b-339a5d8ed640-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.547616 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.547576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9020da08-566c-41f1-bf8b-339a5d8ed640-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.547616 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.547626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9020da08-566c-41f1-bf8b-339a5d8ed640-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.547859 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.547647    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9020da08-566c-41f1-bf8b-339a5d8ed640-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.547859 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.547666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9020da08-566c-41f1-bf8b-339a5d8ed640-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.547859 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.547690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9020da08-566c-41f1-bf8b-339a5d8ed640-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.547859 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.547706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxxvd\" (UniqueName: \"kubernetes.io/projected/9020da08-566c-41f1-bf8b-339a5d8ed640-kube-api-access-bxxvd\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.547859 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.547731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9020da08-566c-41f1-bf8b-339a5d8ed640-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.547859 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.547758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9020da08-566c-41f1-bf8b-339a5d8ed640-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.547859 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.547782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9020da08-566c-41f1-bf8b-339a5d8ed640-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.548219 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.548025    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9020da08-566c-41f1-bf8b-339a5d8ed640-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.548219 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.548037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9020da08-566c-41f1-bf8b-339a5d8ed640-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.548219 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.548105    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9020da08-566c-41f1-bf8b-339a5d8ed640-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.548457 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.548434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9020da08-566c-41f1-bf8b-339a5d8ed640-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.548765 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.548745    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9020da08-566c-41f1-bf8b-339a5d8ed640-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.550139 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.550119    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9020da08-566c-41f1-bf8b-339a5d8ed640-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.550405 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.550383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9020da08-566c-41f1-bf8b-339a5d8ed640-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.592749 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.592720    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxxvd\" (UniqueName: \"kubernetes.io/projected/9020da08-566c-41f1-bf8b-339a5d8ed640-kube-api-access-bxxvd\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.593092 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.593070    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9020da08-566c-41f1-bf8b-339a5d8ed640-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-vqq7v\" (UID: \"9020da08-566c-41f1-bf8b-339a5d8ed640\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.684210 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.684093    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:46.837457 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:46.837429    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"]
Jul 31 14:55:46.841033 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:55:46.841006    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9020da08_566c_41f1_bf8b_339a5d8ed640.slice/crio-50b577fef7eef4dd24f78f2a603e71e7f0d5a7efe51f7668c5c75b2b9150e566 WatchSource:0}: Error finding container 50b577fef7eef4dd24f78f2a603e71e7f0d5a7efe51f7668c5c75b2b9150e566: Status 404 returned error can't find the container with id 50b577fef7eef4dd24f78f2a603e71e7f0d5a7efe51f7668c5c75b2b9150e566
Jul 31 14:55:47.106182 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:47.106150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v" event={"ID":"9020da08-566c-41f1-bf8b-339a5d8ed640","Type":"ContainerStarted","Data":"50b577fef7eef4dd24f78f2a603e71e7f0d5a7efe51f7668c5c75b2b9150e566"}
Jul 31 14:55:49.479582 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:49.479537    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 14:55:49.479821 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:49.479631    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 14:55:49.479821 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:49.479668    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 14:55:50.119076 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:50.119038    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v" event={"ID":"9020da08-566c-41f1-bf8b-339a5d8ed640","Type":"ContainerStarted","Data":"a605110fb5a49b2555d075edd8ea4cf0de128e27595e2576719e5b074627dbf3"}
Jul 31 14:55:50.149663 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:50.149611    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v" podStartSLOduration=1.513113754 podStartE2EDuration="4.149596684s" totalImagesPullingTime="2.63648293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:55:50.149038832 +0000 UTC m=+448.158250701" watchObservedRunningTime="2026-07-31 14:55:50.149596684 +0000 UTC m=+448.158808615"
Jul 31 14:55:50.684767 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:50.684721    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:50.690025 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:50.690001    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:51.122903 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:51.122873    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:51.123735 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:51.123712    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 14:55:56.664195 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.664155    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"]
Jul 31 14:55:56.667450 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.667431    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"
Jul 31 14:55:56.671761 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.671740    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 31 14:55:56.672028 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.672002    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 31 14:55:56.672028 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.672029    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4hjz4\""
Jul 31 14:55:56.672210 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.672004    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 31 14:55:56.684731 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.684709    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"]
Jul 31 14:55:56.729189 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.729161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d9c57708-6c5a-4258-9d55-bd3b472e825d-manager-config\") pod \"lws-controller-manager-56854c75c8-2j2t9\" (UID: \"d9c57708-6c5a-4258-9d55-bd3b472e825d\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"
Jul 31 14:55:56.729339 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.729196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9c57708-6c5a-4258-9d55-bd3b472e825d-metrics-cert\") pod \"lws-controller-manager-56854c75c8-2j2t9\" (UID: \"d9c57708-6c5a-4258-9d55-bd3b472e825d\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"
Jul 31 14:55:56.729339 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.729270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9c57708-6c5a-4258-9d55-bd3b472e825d-cert\") pod \"lws-controller-manager-56854c75c8-2j2t9\" (UID: \"d9c57708-6c5a-4258-9d55-bd3b472e825d\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"
Jul 31 14:55:56.729339 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.729322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcf96\" (UniqueName: \"kubernetes.io/projected/d9c57708-6c5a-4258-9d55-bd3b472e825d-kube-api-access-qcf96\") pod \"lws-controller-manager-56854c75c8-2j2t9\" (UID: \"d9c57708-6c5a-4258-9d55-bd3b472e825d\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"
Jul 31 14:55:56.830289 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.830245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcf96\" (UniqueName: \"kubernetes.io/projected/d9c57708-6c5a-4258-9d55-bd3b472e825d-kube-api-access-qcf96\") pod \"lws-controller-manager-56854c75c8-2j2t9\" (UID: \"d9c57708-6c5a-4258-9d55-bd3b472e825d\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"
Jul 31 14:55:56.830289 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.830298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d9c57708-6c5a-4258-9d55-bd3b472e825d-manager-config\") pod \"lws-controller-manager-56854c75c8-2j2t9\" (UID: \"d9c57708-6c5a-4258-9d55-bd3b472e825d\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"
Jul 31 14:55:56.830557 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.830316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9c57708-6c5a-4258-9d55-bd3b472e825d-metrics-cert\") pod \"lws-controller-manager-56854c75c8-2j2t9\" (UID: \"d9c57708-6c5a-4258-9d55-bd3b472e825d\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"
Jul 31 14:55:56.830557 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.830376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9c57708-6c5a-4258-9d55-bd3b472e825d-cert\") pod \"lws-controller-manager-56854c75c8-2j2t9\" (UID: \"d9c57708-6c5a-4258-9d55-bd3b472e825d\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"
Jul 31 14:55:56.830928 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.830905    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d9c57708-6c5a-4258-9d55-bd3b472e825d-manager-config\") pod \"lws-controller-manager-56854c75c8-2j2t9\" (UID: \"d9c57708-6c5a-4258-9d55-bd3b472e825d\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"
Jul 31 14:55:56.832881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.832856    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9c57708-6c5a-4258-9d55-bd3b472e825d-metrics-cert\") pod \"lws-controller-manager-56854c75c8-2j2t9\" (UID: \"d9c57708-6c5a-4258-9d55-bd3b472e825d\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"
Jul 31 14:55:56.832967 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.832935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9c57708-6c5a-4258-9d55-bd3b472e825d-cert\") pod \"lws-controller-manager-56854c75c8-2j2t9\" (UID: \"d9c57708-6c5a-4258-9d55-bd3b472e825d\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"
Jul 31 14:55:56.840549 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.840522    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcf96\" (UniqueName: \"kubernetes.io/projected/d9c57708-6c5a-4258-9d55-bd3b472e825d-kube-api-access-qcf96\") pod \"lws-controller-manager-56854c75c8-2j2t9\" (UID: \"d9c57708-6c5a-4258-9d55-bd3b472e825d\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"
Jul 31 14:55:56.975918 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:56.975887    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"
Jul 31 14:55:57.158235 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:57.158202    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"]
Jul 31 14:55:57.161982 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:55:57.161948    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c57708_6c5a_4258_9d55_bd3b472e825d.slice/crio-9a2f6ec8db5408c4ff623298047cfa09907530704b3845468cbb3930ce43afa8 WatchSource:0}: Error finding container 9a2f6ec8db5408c4ff623298047cfa09907530704b3845468cbb3930ce43afa8: Status 404 returned error can't find the container with id 9a2f6ec8db5408c4ff623298047cfa09907530704b3845468cbb3930ce43afa8
Jul 31 14:55:58.148559 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:58.148526    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9" event={"ID":"d9c57708-6c5a-4258-9d55-bd3b472e825d","Type":"ContainerStarted","Data":"9a2f6ec8db5408c4ff623298047cfa09907530704b3845468cbb3930ce43afa8"}
Jul 31 14:55:59.153664 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:59.153627    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9" event={"ID":"d9c57708-6c5a-4258-9d55-bd3b472e825d","Type":"ContainerStarted","Data":"e38ab0846ec2cf37753ebe3f799c86ce9b69d642fa10fd8ac0a203df1d7fe0c2"}
Jul 31 14:55:59.154132 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:59.153755    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"
Jul 31 14:55:59.176849 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:55:59.176798    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9" podStartSLOduration=1.377348993 podStartE2EDuration="3.176784786s" totalImagesPullingTime="1.799435793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:55:59.17645569 +0000 UTC m=+457.185667559" watchObservedRunningTime="2026-07-31 14:55:59.176784786 +0000 UTC m=+457.185996655"
Jul 31 14:56:10.159745 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:10.159713    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-56854c75c8-2j2t9"
Jul 31 14:56:24.984629 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:24.984594    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp"]
Jul 31 14:56:24.994652 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:24.994628    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp"
Jul 31 14:56:24.998480 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:24.998454    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 31 14:56:24.998811 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:24.998795    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 31 14:56:25.001987 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:25.001966    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 31 14:56:25.002555 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:25.002536    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ffvtl\""
Jul 31 14:56:25.002647 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:25.002552    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 31 14:56:25.003480 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:25.003461    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp"]
Jul 31 14:56:25.144651 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:25.144608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2b4e9d1a-343e-45a6-b738-02afc1fa3a1d-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-kwzdp\" (UID: \"2b4e9d1a-343e-45a6-b738-02afc1fa3a1d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp"
Jul 31 14:56:25.144651 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:25.144654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b4e9d1a-343e-45a6-b738-02afc1fa3a1d-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-kwzdp\" (UID: \"2b4e9d1a-343e-45a6-b738-02afc1fa3a1d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp"
Jul 31 14:56:25.144866 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:25.144680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8shx7\" (UniqueName: \"kubernetes.io/projected/2b4e9d1a-343e-45a6-b738-02afc1fa3a1d-kube-api-access-8shx7\") pod \"kuadrant-console-plugin-598f86468b-kwzdp\" (UID: \"2b4e9d1a-343e-45a6-b738-02afc1fa3a1d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp"
Jul 31 14:56:25.245885 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:25.245807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b4e9d1a-343e-45a6-b738-02afc1fa3a1d-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-kwzdp\" (UID: \"2b4e9d1a-343e-45a6-b738-02afc1fa3a1d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp"
Jul 31 14:56:25.245885 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:25.245842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8shx7\" (UniqueName: \"kubernetes.io/projected/2b4e9d1a-343e-45a6-b738-02afc1fa3a1d-kube-api-access-8shx7\") pod \"kuadrant-console-plugin-598f86468b-kwzdp\" (UID: \"2b4e9d1a-343e-45a6-b738-02afc1fa3a1d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp"
Jul 31 14:56:25.246066 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:25.245886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2b4e9d1a-343e-45a6-b738-02afc1fa3a1d-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-kwzdp\" (UID: \"2b4e9d1a-343e-45a6-b738-02afc1fa3a1d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp"
Jul 31 14:56:25.246066 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:56:25.245967    2578 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 31 14:56:25.246066 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:56:25.246034    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b4e9d1a-343e-45a6-b738-02afc1fa3a1d-plugin-serving-cert podName:2b4e9d1a-343e-45a6-b738-02afc1fa3a1d nodeName:}" failed. No retries permitted until 2026-07-31 14:56:25.74601466 +0000 UTC m=+483.755226513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/2b4e9d1a-343e-45a6-b738-02afc1fa3a1d-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-kwzdp" (UID: "2b4e9d1a-343e-45a6-b738-02afc1fa3a1d") : secret "plugin-serving-cert" not found
Jul 31 14:56:25.246570 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:25.246552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2b4e9d1a-343e-45a6-b738-02afc1fa3a1d-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-kwzdp\" (UID: \"2b4e9d1a-343e-45a6-b738-02afc1fa3a1d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp"
Jul 31 14:56:25.259594 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:25.259573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8shx7\" (UniqueName: \"kubernetes.io/projected/2b4e9d1a-343e-45a6-b738-02afc1fa3a1d-kube-api-access-8shx7\") pod \"kuadrant-console-plugin-598f86468b-kwzdp\" (UID: \"2b4e9d1a-343e-45a6-b738-02afc1fa3a1d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp"
Jul 31 14:56:25.750867 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:25.750832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b4e9d1a-343e-45a6-b738-02afc1fa3a1d-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-kwzdp\" (UID: \"2b4e9d1a-343e-45a6-b738-02afc1fa3a1d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp"
Jul 31 14:56:25.751050 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:56:25.750983    2578 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 31 14:56:25.751111 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:56:25.751068    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b4e9d1a-343e-45a6-b738-02afc1fa3a1d-plugin-serving-cert podName:2b4e9d1a-343e-45a6-b738-02afc1fa3a1d nodeName:}" failed. No retries permitted until 2026-07-31 14:56:26.751047528 +0000 UTC m=+484.760259375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/2b4e9d1a-343e-45a6-b738-02afc1fa3a1d-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-kwzdp" (UID: "2b4e9d1a-343e-45a6-b738-02afc1fa3a1d") : secret "plugin-serving-cert" not found
Jul 31 14:56:26.760578 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:26.760541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b4e9d1a-343e-45a6-b738-02afc1fa3a1d-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-kwzdp\" (UID: \"2b4e9d1a-343e-45a6-b738-02afc1fa3a1d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp"
Jul 31 14:56:26.762865 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:26.762842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b4e9d1a-343e-45a6-b738-02afc1fa3a1d-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-kwzdp\" (UID: \"2b4e9d1a-343e-45a6-b738-02afc1fa3a1d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp"
Jul 31 14:56:26.804538 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:26.804499    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp"
Jul 31 14:56:26.931332 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:26.931300    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp"]
Jul 31 14:56:26.934550 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:56:26.934526    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b4e9d1a_343e_45a6_b738_02afc1fa3a1d.slice/crio-f3abebdff83db4b10b6d47e1568d339d36bd68db7143c4d4ebce59d0ecebe604 WatchSource:0}: Error finding container f3abebdff83db4b10b6d47e1568d339d36bd68db7143c4d4ebce59d0ecebe604: Status 404 returned error can't find the container with id f3abebdff83db4b10b6d47e1568d339d36bd68db7143c4d4ebce59d0ecebe604
Jul 31 14:56:27.250639 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:27.250602    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp" event={"ID":"2b4e9d1a-343e-45a6-b738-02afc1fa3a1d","Type":"ContainerStarted","Data":"f3abebdff83db4b10b6d47e1568d339d36bd68db7143c4d4ebce59d0ecebe604"}
Jul 31 14:56:32.273221 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:32.273179    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp" event={"ID":"2b4e9d1a-343e-45a6-b738-02afc1fa3a1d","Type":"ContainerStarted","Data":"574ad65e48077af6a8d82d75303e4cbdcd6945a2ad979de03e4cefda450686ef"}
Jul 31 14:56:32.291557 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:56:32.291511    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-kwzdp" podStartSLOduration=3.716312017 podStartE2EDuration="8.291497191s" totalImagesPullingTime="4.575185174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:56:32.290762728 +0000 UTC m=+490.299974598" watchObservedRunningTime="2026-07-31 14:56:32.291497191 +0000 UTC m=+490.300709060"
Jul 31 14:57:07.455970 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:07.455935    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-62r5s"]
Jul 31 14:57:07.475930 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:07.475894    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-62r5s"]
Jul 31 14:57:07.476070 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:07.475992    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-62r5s"
Jul 31 14:57:07.479522 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:07.479499    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lhsb8\""
Jul 31 14:57:07.564877 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:07.564842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t96nh\" (UniqueName: \"kubernetes.io/projected/3c4f26b8-9b52-4c33-9e8b-5703125af07f-kube-api-access-t96nh\") pod \"authorino-b54df94b4-62r5s\" (UID: \"3c4f26b8-9b52-4c33-9e8b-5703125af07f\") " pod="kuadrant-system/authorino-b54df94b4-62r5s"
Jul 31 14:57:07.626168 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:07.626116    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-9qqvl"]
Jul 31 14:57:07.629343 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:07.629327    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-9qqvl"
Jul 31 14:57:07.638034 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:07.638011    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-9qqvl"]
Jul 31 14:57:07.665546 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:07.665512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxgkl\" (UniqueName: \"kubernetes.io/projected/d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244-kube-api-access-wxgkl\") pod \"authorino-c548b5d7d-9qqvl\" (UID: \"d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244\") " pod="kuadrant-system/authorino-c548b5d7d-9qqvl"
Jul 31 14:57:07.665701 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:07.665567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t96nh\" (UniqueName: \"kubernetes.io/projected/3c4f26b8-9b52-4c33-9e8b-5703125af07f-kube-api-access-t96nh\") pod \"authorino-b54df94b4-62r5s\" (UID: \"3c4f26b8-9b52-4c33-9e8b-5703125af07f\") " pod="kuadrant-system/authorino-b54df94b4-62r5s"
Jul 31 14:57:07.686765 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:07.686738    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t96nh\" (UniqueName: \"kubernetes.io/projected/3c4f26b8-9b52-4c33-9e8b-5703125af07f-kube-api-access-t96nh\") pod \"authorino-b54df94b4-62r5s\" (UID: \"3c4f26b8-9b52-4c33-9e8b-5703125af07f\") " pod="kuadrant-system/authorino-b54df94b4-62r5s"
Jul 31 14:57:07.765971 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:07.765933    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxgkl\" (UniqueName: \"kubernetes.io/projected/d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244-kube-api-access-wxgkl\") pod \"authorino-c548b5d7d-9qqvl\" (UID: \"d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244\") " pod="kuadrant-system/authorino-c548b5d7d-9qqvl"
Jul 31 14:57:07.775846 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:07.775815    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxgkl\" (UniqueName: \"kubernetes.io/projected/d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244-kube-api-access-wxgkl\") pod \"authorino-c548b5d7d-9qqvl\" (UID: \"d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244\") " pod="kuadrant-system/authorino-c548b5d7d-9qqvl"
Jul 31 14:57:07.784633 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:07.784610    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-62r5s"
Jul 31 14:57:07.915033 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:07.915003    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-62r5s"]
Jul 31 14:57:07.917559 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:57:07.917530    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c4f26b8_9b52_4c33_9e8b_5703125af07f.slice/crio-fd539d8add4987bbe98ad4de50b602e48b39d4a59bb479f6295e0f6369f10598 WatchSource:0}: Error finding container fd539d8add4987bbe98ad4de50b602e48b39d4a59bb479f6295e0f6369f10598: Status 404 returned error can't find the container with id fd539d8add4987bbe98ad4de50b602e48b39d4a59bb479f6295e0f6369f10598
Jul 31 14:57:07.938297 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:07.938270    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-9qqvl"
Jul 31 14:57:08.060199 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:08.060175    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-9qqvl"]
Jul 31 14:57:08.062406 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:57:08.062384    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd23d5fbc_1c01_4cb4_b85b_3fb2f0be8244.slice/crio-b802760fc7cd7a20e6424c3b2fb74f1fb642ff1991df5179d811979c37926a25 WatchSource:0}: Error finding container b802760fc7cd7a20e6424c3b2fb74f1fb642ff1991df5179d811979c37926a25: Status 404 returned error can't find the container with id b802760fc7cd7a20e6424c3b2fb74f1fb642ff1991df5179d811979c37926a25
Jul 31 14:57:08.404392 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:08.404298    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-62r5s" event={"ID":"3c4f26b8-9b52-4c33-9e8b-5703125af07f","Type":"ContainerStarted","Data":"fd539d8add4987bbe98ad4de50b602e48b39d4a59bb479f6295e0f6369f10598"}
Jul 31 14:57:08.405389 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:08.405366    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-9qqvl" event={"ID":"d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244","Type":"ContainerStarted","Data":"b802760fc7cd7a20e6424c3b2fb74f1fb642ff1991df5179d811979c37926a25"}
Jul 31 14:57:11.421269 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:11.421233    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-62r5s" event={"ID":"3c4f26b8-9b52-4c33-9e8b-5703125af07f","Type":"ContainerStarted","Data":"466c208cd5296febecbb774f0bd516f2cfd514a76598d264b3a7ee2e58f40a80"}
Jul 31 14:57:11.422546 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:11.422522    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-9qqvl" event={"ID":"d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244","Type":"ContainerStarted","Data":"84ed47c2edfb7447de99a6b64dd6d8f093843e76003cc1a20daa3cd764956607"}
Jul 31 14:57:11.439246 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:11.439203    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-62r5s" podStartSLOduration=1.65763982 podStartE2EDuration="4.439187313s" totalImagesPullingTime="2.781547493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:57:11.438612274 +0000 UTC m=+529.447824143" watchObservedRunningTime="2026-07-31 14:57:11.439187313 +0000 UTC m=+529.448399182"
Jul 31 14:57:11.462382 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:11.458615    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-9qqvl" podStartSLOduration=1.810356399 podStartE2EDuration="4.458595699s" totalImagesPullingTime="2.6482393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:57:11.456396612 +0000 UTC m=+529.465608483" watchObservedRunningTime="2026-07-31 14:57:11.458595699 +0000 UTC m=+529.467807569"
Jul 31 14:57:11.540717 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:11.540685    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-62r5s"]
Jul 31 14:57:13.429436 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:13.429396    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-62r5s" podUID="3c4f26b8-9b52-4c33-9e8b-5703125af07f" containerName="authorino" containerID="cri-o://466c208cd5296febecbb774f0bd516f2cfd514a76598d264b3a7ee2e58f40a80" gracePeriod=30
Jul 31 14:57:13.670391 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:13.670360    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-62r5s"
Jul 31 14:57:13.717819 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:13.717735    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t96nh\" (UniqueName: \"kubernetes.io/projected/3c4f26b8-9b52-4c33-9e8b-5703125af07f-kube-api-access-t96nh\") pod \"3c4f26b8-9b52-4c33-9e8b-5703125af07f\" (UID: \"3c4f26b8-9b52-4c33-9e8b-5703125af07f\") "
Jul 31 14:57:13.719881 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:13.719859    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4f26b8-9b52-4c33-9e8b-5703125af07f-kube-api-access-t96nh" (OuterVolumeSpecName: "kube-api-access-t96nh") pod "3c4f26b8-9b52-4c33-9e8b-5703125af07f" (UID: "3c4f26b8-9b52-4c33-9e8b-5703125af07f"). InnerVolumeSpecName "kube-api-access-t96nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:57:13.818649 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:13.818605    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t96nh\" (UniqueName: \"kubernetes.io/projected/3c4f26b8-9b52-4c33-9e8b-5703125af07f-kube-api-access-t96nh\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 14:57:14.433871 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:14.433795    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c4f26b8-9b52-4c33-9e8b-5703125af07f" containerID="466c208cd5296febecbb774f0bd516f2cfd514a76598d264b3a7ee2e58f40a80" exitCode=0
Jul 31 14:57:14.433871 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:14.433840    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-62r5s"
Jul 31 14:57:14.433871 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:14.433858    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-62r5s" event={"ID":"3c4f26b8-9b52-4c33-9e8b-5703125af07f","Type":"ContainerDied","Data":"466c208cd5296febecbb774f0bd516f2cfd514a76598d264b3a7ee2e58f40a80"}
Jul 31 14:57:14.434325 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:14.433893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-62r5s" event={"ID":"3c4f26b8-9b52-4c33-9e8b-5703125af07f","Type":"ContainerDied","Data":"fd539d8add4987bbe98ad4de50b602e48b39d4a59bb479f6295e0f6369f10598"}
Jul 31 14:57:14.434325 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:14.433913    2578 scope.go:117] "RemoveContainer" containerID="466c208cd5296febecbb774f0bd516f2cfd514a76598d264b3a7ee2e58f40a80"
Jul 31 14:57:14.442576 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:14.442559    2578 scope.go:117] "RemoveContainer" containerID="466c208cd5296febecbb774f0bd516f2cfd514a76598d264b3a7ee2e58f40a80"
Jul 31 14:57:14.442842 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:57:14.442823    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466c208cd5296febecbb774f0bd516f2cfd514a76598d264b3a7ee2e58f40a80\": container with ID starting with 466c208cd5296febecbb774f0bd516f2cfd514a76598d264b3a7ee2e58f40a80 not found: ID does not exist" containerID="466c208cd5296febecbb774f0bd516f2cfd514a76598d264b3a7ee2e58f40a80"
Jul 31 14:57:14.442920 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:14.442855    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466c208cd5296febecbb774f0bd516f2cfd514a76598d264b3a7ee2e58f40a80"} err="failed to get container status \"466c208cd5296febecbb774f0bd516f2cfd514a76598d264b3a7ee2e58f40a80\": rpc error: code = NotFound desc = could not find container \"466c208cd5296febecbb774f0bd516f2cfd514a76598d264b3a7ee2e58f40a80\": container with ID starting with 466c208cd5296febecbb774f0bd516f2cfd514a76598d264b3a7ee2e58f40a80 not found: ID does not exist"
Jul 31 14:57:14.456561 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:14.456537    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-62r5s"]
Jul 31 14:57:14.463327 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:14.463301    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-62r5s"]
Jul 31 14:57:14.488470 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:14.488443    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c4f26b8-9b52-4c33-9e8b-5703125af07f" path="/var/lib/kubelet/pods/3c4f26b8-9b52-4c33-9e8b-5703125af07f/volumes"
Jul 31 14:57:30.326939 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:30.326910    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cf4765b85-nc4f9"]
Jul 31 14:57:30.327312 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:30.327200    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c4f26b8-9b52-4c33-9e8b-5703125af07f" containerName="authorino"
Jul 31 14:57:30.327312 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:30.327213    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c4f26b8-9b52-4c33-9e8b-5703125af07f" containerName="authorino"
Jul 31 14:57:30.327312 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:30.327279    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c4f26b8-9b52-4c33-9e8b-5703125af07f" containerName="authorino"
Jul 31 14:57:30.331171 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:30.331156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-nc4f9"
Jul 31 14:57:30.334577 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:30.334561    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 31 14:57:30.340164 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:30.340141    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-nc4f9"]
Jul 31 14:57:30.448001 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:30.447957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d2cbc9d-b1de-45c3-bbf7-6af3b67d8b4f-tls-cert\") pod \"authorino-7cf4765b85-nc4f9\" (UID: \"5d2cbc9d-b1de-45c3-bbf7-6af3b67d8b4f\") " pod="kuadrant-system/authorino-7cf4765b85-nc4f9"
Jul 31 14:57:30.448173 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:30.448018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72tjr\" (UniqueName: \"kubernetes.io/projected/5d2cbc9d-b1de-45c3-bbf7-6af3b67d8b4f-kube-api-access-72tjr\") pod \"authorino-7cf4765b85-nc4f9\" (UID: \"5d2cbc9d-b1de-45c3-bbf7-6af3b67d8b4f\") " pod="kuadrant-system/authorino-7cf4765b85-nc4f9"
Jul 31 14:57:30.549469 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:30.549416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d2cbc9d-b1de-45c3-bbf7-6af3b67d8b4f-tls-cert\") pod \"authorino-7cf4765b85-nc4f9\" (UID: \"5d2cbc9d-b1de-45c3-bbf7-6af3b67d8b4f\") " pod="kuadrant-system/authorino-7cf4765b85-nc4f9"
Jul 31 14:57:30.549651 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:30.549493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72tjr\" (UniqueName: \"kubernetes.io/projected/5d2cbc9d-b1de-45c3-bbf7-6af3b67d8b4f-kube-api-access-72tjr\") pod \"authorino-7cf4765b85-nc4f9\" (UID: \"5d2cbc9d-b1de-45c3-bbf7-6af3b67d8b4f\") " pod="kuadrant-system/authorino-7cf4765b85-nc4f9"
Jul 31 14:57:30.551826 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:30.551799    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d2cbc9d-b1de-45c3-bbf7-6af3b67d8b4f-tls-cert\") pod \"authorino-7cf4765b85-nc4f9\" (UID: \"5d2cbc9d-b1de-45c3-bbf7-6af3b67d8b4f\") " pod="kuadrant-system/authorino-7cf4765b85-nc4f9"
Jul 31 14:57:30.559543 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:30.559515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72tjr\" (UniqueName: \"kubernetes.io/projected/5d2cbc9d-b1de-45c3-bbf7-6af3b67d8b4f-kube-api-access-72tjr\") pod \"authorino-7cf4765b85-nc4f9\" (UID: \"5d2cbc9d-b1de-45c3-bbf7-6af3b67d8b4f\") " pod="kuadrant-system/authorino-7cf4765b85-nc4f9"
Jul 31 14:57:30.639909 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:30.639828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-nc4f9"
Jul 31 14:57:30.769173 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:30.769140    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-nc4f9"]
Jul 31 14:57:30.773189 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:57:30.773163    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d2cbc9d_b1de_45c3_bbf7_6af3b67d8b4f.slice/crio-938e346c1cc4ed91493d0550c2f5d555096c09d006d3ab03d6d51db31c2d5dd0 WatchSource:0}: Error finding container 938e346c1cc4ed91493d0550c2f5d555096c09d006d3ab03d6d51db31c2d5dd0: Status 404 returned error can't find the container with id 938e346c1cc4ed91493d0550c2f5d555096c09d006d3ab03d6d51db31c2d5dd0
Jul 31 14:57:31.492430 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:31.492396    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-nc4f9" event={"ID":"5d2cbc9d-b1de-45c3-bbf7-6af3b67d8b4f","Type":"ContainerStarted","Data":"938e346c1cc4ed91493d0550c2f5d555096c09d006d3ab03d6d51db31c2d5dd0"}
Jul 31 14:57:33.500404 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:33.500345    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-nc4f9" event={"ID":"5d2cbc9d-b1de-45c3-bbf7-6af3b67d8b4f","Type":"ContainerStarted","Data":"d324cffbbfd33ad1975c71a15088742d1a3d038f4ff170246b8552c316f4a645"}
Jul 31 14:57:33.518323 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:33.518275    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cf4765b85-nc4f9" podStartSLOduration=1.376090891 podStartE2EDuration="3.518261269s" totalImagesPullingTime="2.142170378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:57:33.517264311 +0000 UTC m=+551.526476180" watchObservedRunningTime="2026-07-31 14:57:33.518261269 +0000 UTC m=+551.527473138"
Jul 31 14:57:33.546587 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:33.546555    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-9qqvl"]
Jul 31 14:57:33.546795 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:33.546769    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-9qqvl" podUID="d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244" containerName="authorino" containerID="cri-o://84ed47c2edfb7447de99a6b64dd6d8f093843e76003cc1a20daa3cd764956607" gracePeriod=30
Jul 31 14:57:33.782618 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:33.782595    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-9qqvl"
Jul 31 14:57:33.876314 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:33.876282    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxgkl\" (UniqueName: \"kubernetes.io/projected/d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244-kube-api-access-wxgkl\") pod \"d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244\" (UID: \"d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244\") "
Jul 31 14:57:33.878260 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:33.878236    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244-kube-api-access-wxgkl" (OuterVolumeSpecName: "kube-api-access-wxgkl") pod "d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244" (UID: "d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244"). InnerVolumeSpecName "kube-api-access-wxgkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:57:33.976763 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:33.976727    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxgkl\" (UniqueName: \"kubernetes.io/projected/d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244-kube-api-access-wxgkl\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 14:57:34.505034 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:34.505003    2578 generic.go:356] "Generic (PLEG): container finished" podID="d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244" containerID="84ed47c2edfb7447de99a6b64dd6d8f093843e76003cc1a20daa3cd764956607" exitCode=0
Jul 31 14:57:34.505430 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:34.505042    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-9qqvl"
Jul 31 14:57:34.505430 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:34.505072    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-9qqvl" event={"ID":"d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244","Type":"ContainerDied","Data":"84ed47c2edfb7447de99a6b64dd6d8f093843e76003cc1a20daa3cd764956607"}
Jul 31 14:57:34.505430 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:34.505100    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-9qqvl" event={"ID":"d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244","Type":"ContainerDied","Data":"b802760fc7cd7a20e6424c3b2fb74f1fb642ff1991df5179d811979c37926a25"}
Jul 31 14:57:34.505430 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:34.505117    2578 scope.go:117] "RemoveContainer" containerID="84ed47c2edfb7447de99a6b64dd6d8f093843e76003cc1a20daa3cd764956607"
Jul 31 14:57:34.512732 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:34.512702    2578 scope.go:117] "RemoveContainer" containerID="84ed47c2edfb7447de99a6b64dd6d8f093843e76003cc1a20daa3cd764956607"
Jul 31 14:57:34.512992 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:57:34.512971    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ed47c2edfb7447de99a6b64dd6d8f093843e76003cc1a20daa3cd764956607\": container with ID starting with 84ed47c2edfb7447de99a6b64dd6d8f093843e76003cc1a20daa3cd764956607 not found: ID does not exist" containerID="84ed47c2edfb7447de99a6b64dd6d8f093843e76003cc1a20daa3cd764956607"
Jul 31 14:57:34.512992 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:34.513000    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ed47c2edfb7447de99a6b64dd6d8f093843e76003cc1a20daa3cd764956607"} err="failed to get container status \"84ed47c2edfb7447de99a6b64dd6d8f093843e76003cc1a20daa3cd764956607\": rpc error: code = NotFound desc = could not find container \"84ed47c2edfb7447de99a6b64dd6d8f093843e76003cc1a20daa3cd764956607\": container with ID starting with 84ed47c2edfb7447de99a6b64dd6d8f093843e76003cc1a20daa3cd764956607 not found: ID does not exist"
Jul 31 14:57:34.522684 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:34.522663    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-9qqvl"]
Jul 31 14:57:34.526699 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:34.526673    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-9qqvl"]
Jul 31 14:57:36.489225 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:57:36.489194    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244" path="/var/lib/kubelet/pods/d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244/volumes"
Jul 31 14:59:27.836826 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:27.836788    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-8588677cfb-krprn"]
Jul 31 14:59:27.837263 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:27.837040    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244" containerName="authorino"
Jul 31 14:59:27.837263 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:27.837051    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244" containerName="authorino"
Jul 31 14:59:27.837263 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:27.837100    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d23d5fbc-1c01-4cb4-b85b-3fb2f0be8244" containerName="authorino"
Jul 31 14:59:27.839734 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:27.839719    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-krprn"
Jul 31 14:59:27.842637 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:27.842610    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 31 14:59:27.843897 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:27.843877    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Jul 31 14:59:27.843983 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:27.843893    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 31 14:59:27.846944 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:27.846922    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 31 14:59:27.846944 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:27.846932    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-s5mzz\""
Jul 31 14:59:27.861203 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:27.861174    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-krprn"]
Jul 31 14:59:27.926743 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:27.926702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65ecc030-2e12-4227-bdc5-48b75d9bb5a4-metrics-certs\") pod \"odh-model-controller-8588677cfb-krprn\" (UID: \"65ecc030-2e12-4227-bdc5-48b75d9bb5a4\") " pod="kserve/odh-model-controller-8588677cfb-krprn"
Jul 31 14:59:27.926929 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:27.926769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26b7b\" (UniqueName: \"kubernetes.io/projected/65ecc030-2e12-4227-bdc5-48b75d9bb5a4-kube-api-access-26b7b\") pod \"odh-model-controller-8588677cfb-krprn\" (UID: \"65ecc030-2e12-4227-bdc5-48b75d9bb5a4\") " pod="kserve/odh-model-controller-8588677cfb-krprn"
Jul 31 14:59:27.926929 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:27.926818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65ecc030-2e12-4227-bdc5-48b75d9bb5a4-cert\") pod \"odh-model-controller-8588677cfb-krprn\" (UID: \"65ecc030-2e12-4227-bdc5-48b75d9bb5a4\") " pod="kserve/odh-model-controller-8588677cfb-krprn"
Jul 31 14:59:28.027919 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:28.027880    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65ecc030-2e12-4227-bdc5-48b75d9bb5a4-metrics-certs\") pod \"odh-model-controller-8588677cfb-krprn\" (UID: \"65ecc030-2e12-4227-bdc5-48b75d9bb5a4\") " pod="kserve/odh-model-controller-8588677cfb-krprn"
Jul 31 14:59:28.028094 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:28.027937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26b7b\" (UniqueName: \"kubernetes.io/projected/65ecc030-2e12-4227-bdc5-48b75d9bb5a4-kube-api-access-26b7b\") pod \"odh-model-controller-8588677cfb-krprn\" (UID: \"65ecc030-2e12-4227-bdc5-48b75d9bb5a4\") " pod="kserve/odh-model-controller-8588677cfb-krprn"
Jul 31 14:59:28.028094 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:28.027982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65ecc030-2e12-4227-bdc5-48b75d9bb5a4-cert\") pod \"odh-model-controller-8588677cfb-krprn\" (UID: \"65ecc030-2e12-4227-bdc5-48b75d9bb5a4\") " pod="kserve/odh-model-controller-8588677cfb-krprn"
Jul 31 14:59:28.028094 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:59:28.028033    2578 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Jul 31 14:59:28.028203 ip-10-0-133-32 kubenswrapper[2578]: E0731 14:59:28.028118    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65ecc030-2e12-4227-bdc5-48b75d9bb5a4-metrics-certs podName:65ecc030-2e12-4227-bdc5-48b75d9bb5a4 nodeName:}" failed. No retries permitted until 2026-07-31 14:59:28.528100961 +0000 UTC m=+666.537312808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/65ecc030-2e12-4227-bdc5-48b75d9bb5a4-metrics-certs") pod "odh-model-controller-8588677cfb-krprn" (UID: "65ecc030-2e12-4227-bdc5-48b75d9bb5a4") : secret "odh-model-controller-metrics-tls" not found
Jul 31 14:59:28.030558 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:28.030536    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65ecc030-2e12-4227-bdc5-48b75d9bb5a4-cert\") pod \"odh-model-controller-8588677cfb-krprn\" (UID: \"65ecc030-2e12-4227-bdc5-48b75d9bb5a4\") " pod="kserve/odh-model-controller-8588677cfb-krprn"
Jul 31 14:59:28.041046 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:28.041000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26b7b\" (UniqueName: \"kubernetes.io/projected/65ecc030-2e12-4227-bdc5-48b75d9bb5a4-kube-api-access-26b7b\") pod \"odh-model-controller-8588677cfb-krprn\" (UID: \"65ecc030-2e12-4227-bdc5-48b75d9bb5a4\") " pod="kserve/odh-model-controller-8588677cfb-krprn"
Jul 31 14:59:28.531243 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:28.531204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65ecc030-2e12-4227-bdc5-48b75d9bb5a4-metrics-certs\") pod \"odh-model-controller-8588677cfb-krprn\" (UID: \"65ecc030-2e12-4227-bdc5-48b75d9bb5a4\") " pod="kserve/odh-model-controller-8588677cfb-krprn"
Jul 31 14:59:28.533572 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:28.533552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65ecc030-2e12-4227-bdc5-48b75d9bb5a4-metrics-certs\") pod \"odh-model-controller-8588677cfb-krprn\" (UID: \"65ecc030-2e12-4227-bdc5-48b75d9bb5a4\") " pod="kserve/odh-model-controller-8588677cfb-krprn"
Jul 31 14:59:28.750151 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:28.750106    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-krprn"
Jul 31 14:59:28.876024 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:28.876001    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-krprn"]
Jul 31 14:59:28.879391 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:59:28.879363    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ecc030_2e12_4227_bdc5_48b75d9bb5a4.slice/crio-5a25d56ac6e047f0e9ce32e3e520627b3d8196dee044637757c27fc4480e2823 WatchSource:0}: Error finding container 5a25d56ac6e047f0e9ce32e3e520627b3d8196dee044637757c27fc4480e2823: Status 404 returned error can't find the container with id 5a25d56ac6e047f0e9ce32e3e520627b3d8196dee044637757c27fc4480e2823
Jul 31 14:59:28.881216 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:28.881197    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:59:28.909834 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:28.909801    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-krprn" event={"ID":"65ecc030-2e12-4227-bdc5-48b75d9bb5a4","Type":"ContainerStarted","Data":"5a25d56ac6e047f0e9ce32e3e520627b3d8196dee044637757c27fc4480e2823"}
Jul 31 14:59:31.925013 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:31.924923    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-krprn" event={"ID":"65ecc030-2e12-4227-bdc5-48b75d9bb5a4","Type":"ContainerStarted","Data":"25d37294e6d8f4fbf83509a40c698947af4d7bf4afb51c8e7dbd33cfa639471e"}
Jul 31 14:59:31.925459 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:31.925019    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-8588677cfb-krprn"
Jul 31 14:59:31.947713 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:31.947662    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-8588677cfb-krprn" podStartSLOduration=2.329857169 podStartE2EDuration="4.947646863s" totalImagesPullingTime="2.617789694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:59:31.946714258 +0000 UTC m=+669.955926126" watchObservedRunningTime="2026-07-31 14:59:31.947646863 +0000 UTC m=+669.956858731"
Jul 31 14:59:42.930910 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:42.930881    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-8588677cfb-krprn"
Jul 31 14:59:43.827273 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:43.827240    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-kzvnv"]
Jul 31 14:59:43.830578 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:43.830561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kzvnv"
Jul 31 14:59:43.833483 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:43.833459    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-jg8zm\""
Jul 31 14:59:43.833592 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:43.833459    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 31 14:59:43.840107 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:43.840080    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-kzvnv"]
Jul 31 14:59:43.930626 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:43.930587    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3793ccfc-bc3d-4858-80d4-c4a56b09a8fc-model-cache\") pod \"s3-init-kzvnv\" (UID: \"3793ccfc-bc3d-4858-80d4-c4a56b09a8fc\") " pod="kserve/s3-init-kzvnv"
Jul 31 14:59:43.930626 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:43.930633    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dctbr\" (UniqueName: \"kubernetes.io/projected/3793ccfc-bc3d-4858-80d4-c4a56b09a8fc-kube-api-access-dctbr\") pod \"s3-init-kzvnv\" (UID: \"3793ccfc-bc3d-4858-80d4-c4a56b09a8fc\") " pod="kserve/s3-init-kzvnv"
Jul 31 14:59:44.031173 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:44.031136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3793ccfc-bc3d-4858-80d4-c4a56b09a8fc-model-cache\") pod \"s3-init-kzvnv\" (UID: \"3793ccfc-bc3d-4858-80d4-c4a56b09a8fc\") " pod="kserve/s3-init-kzvnv"
Jul 31 14:59:44.031578 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:44.031190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dctbr\" (UniqueName: \"kubernetes.io/projected/3793ccfc-bc3d-4858-80d4-c4a56b09a8fc-kube-api-access-dctbr\") pod \"s3-init-kzvnv\" (UID: \"3793ccfc-bc3d-4858-80d4-c4a56b09a8fc\") " pod="kserve/s3-init-kzvnv"
Jul 31 14:59:44.031578 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:44.031504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3793ccfc-bc3d-4858-80d4-c4a56b09a8fc-model-cache\") pod \"s3-init-kzvnv\" (UID: \"3793ccfc-bc3d-4858-80d4-c4a56b09a8fc\") " pod="kserve/s3-init-kzvnv"
Jul 31 14:59:44.041417 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:44.041390    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dctbr\" (UniqueName: \"kubernetes.io/projected/3793ccfc-bc3d-4858-80d4-c4a56b09a8fc-kube-api-access-dctbr\") pod \"s3-init-kzvnv\" (UID: \"3793ccfc-bc3d-4858-80d4-c4a56b09a8fc\") " pod="kserve/s3-init-kzvnv"
Jul 31 14:59:44.139979 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:44.139890    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kzvnv"
Jul 31 14:59:44.276841 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:44.276815    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-kzvnv"]
Jul 31 14:59:44.279207 ip-10-0-133-32 kubenswrapper[2578]: W0731 14:59:44.279181    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3793ccfc_bc3d_4858_80d4_c4a56b09a8fc.slice/crio-3759672a62e5b0c815c0bedda0f993ccece3dad34660e7dc3ead31f3347e6361 WatchSource:0}: Error finding container 3759672a62e5b0c815c0bedda0f993ccece3dad34660e7dc3ead31f3347e6361: Status 404 returned error can't find the container with id 3759672a62e5b0c815c0bedda0f993ccece3dad34660e7dc3ead31f3347e6361
Jul 31 14:59:44.976421 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:44.976330    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kzvnv" event={"ID":"3793ccfc-bc3d-4858-80d4-c4a56b09a8fc","Type":"ContainerStarted","Data":"3759672a62e5b0c815c0bedda0f993ccece3dad34660e7dc3ead31f3347e6361"}
Jul 31 14:59:47.991228 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:47.991189    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kzvnv" event={"ID":"3793ccfc-bc3d-4858-80d4-c4a56b09a8fc","Type":"ContainerStarted","Data":"11f1a96f3c4ef38dc9c14991a4e41e82e4d99f779e41b72cc050c410fd3da7b7"}
Jul 31 14:59:50.000825 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:50.000793    2578 generic.go:356] "Generic (PLEG): container finished" podID="3793ccfc-bc3d-4858-80d4-c4a56b09a8fc" containerID="11f1a96f3c4ef38dc9c14991a4e41e82e4d99f779e41b72cc050c410fd3da7b7" exitCode=0
Jul 31 14:59:50.001196 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:50.000854    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kzvnv" event={"ID":"3793ccfc-bc3d-4858-80d4-c4a56b09a8fc","Type":"ContainerDied","Data":"11f1a96f3c4ef38dc9c14991a4e41e82e4d99f779e41b72cc050c410fd3da7b7"}
Jul 31 14:59:55.025939 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:55.025901    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kzvnv" event={"ID":"3793ccfc-bc3d-4858-80d4-c4a56b09a8fc","Type":"ContainerStarted","Data":"42f7817e4b1ca37e7bbc89f3f648f3773197ab752cb6be97780507dd72acff22"}
Jul 31 14:59:55.046183 ip-10-0-133-32 kubenswrapper[2578]: I0731 14:59:55.046123    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-kzvnv" podStartSLOduration=2.505612548 podStartE2EDuration="12.046108507s" totalImagesPullingTime="7.495654643s" totalInitContainerRuntime="2.044841316s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:59:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:59:55.044250431 +0000 UTC m=+693.053462302" watchObservedRunningTime="2026-07-31 14:59:55.046108507 +0000 UTC m=+693.055320376"
Jul 31 15:00:59.255791 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:00:59.255756    2578 generic.go:356] "Generic (PLEG): container finished" podID="3793ccfc-bc3d-4858-80d4-c4a56b09a8fc" containerID="42f7817e4b1ca37e7bbc89f3f648f3773197ab752cb6be97780507dd72acff22" exitCode=0
Jul 31 15:00:59.256220 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:00:59.255807    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kzvnv" event={"ID":"3793ccfc-bc3d-4858-80d4-c4a56b09a8fc","Type":"ContainerDied","Data":"42f7817e4b1ca37e7bbc89f3f648f3773197ab752cb6be97780507dd72acff22"}
Jul 31 15:01:00.389922 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:00.389899    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kzvnv"
Jul 31 15:01:00.421084 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:00.421060    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3793ccfc-bc3d-4858-80d4-c4a56b09a8fc-model-cache\") pod \"3793ccfc-bc3d-4858-80d4-c4a56b09a8fc\" (UID: \"3793ccfc-bc3d-4858-80d4-c4a56b09a8fc\") "
Jul 31 15:01:00.421203 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:00.421116    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dctbr\" (UniqueName: \"kubernetes.io/projected/3793ccfc-bc3d-4858-80d4-c4a56b09a8fc-kube-api-access-dctbr\") pod \"3793ccfc-bc3d-4858-80d4-c4a56b09a8fc\" (UID: \"3793ccfc-bc3d-4858-80d4-c4a56b09a8fc\") "
Jul 31 15:01:00.423245 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:00.423220    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3793ccfc-bc3d-4858-80d4-c4a56b09a8fc-kube-api-access-dctbr" (OuterVolumeSpecName: "kube-api-access-dctbr") pod "3793ccfc-bc3d-4858-80d4-c4a56b09a8fc" (UID: "3793ccfc-bc3d-4858-80d4-c4a56b09a8fc"). InnerVolumeSpecName "kube-api-access-dctbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:01:00.440680 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:00.440654    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3793ccfc-bc3d-4858-80d4-c4a56b09a8fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "3793ccfc-bc3d-4858-80d4-c4a56b09a8fc" (UID: "3793ccfc-bc3d-4858-80d4-c4a56b09a8fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:01:00.522215 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:00.522147    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dctbr\" (UniqueName: \"kubernetes.io/projected/3793ccfc-bc3d-4858-80d4-c4a56b09a8fc-kube-api-access-dctbr\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:01:00.522215 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:00.522175    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3793ccfc-bc3d-4858-80d4-c4a56b09a8fc-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:01:01.264129 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:01.264100    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kzvnv"
Jul 31 15:01:01.264129 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:01.264113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kzvnv" event={"ID":"3793ccfc-bc3d-4858-80d4-c4a56b09a8fc","Type":"ContainerDied","Data":"3759672a62e5b0c815c0bedda0f993ccece3dad34660e7dc3ead31f3347e6361"}
Jul 31 15:01:01.264379 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:01.264145    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3759672a62e5b0c815c0bedda0f993ccece3dad34660e7dc3ead31f3347e6361"
Jul 31 15:01:24.005988 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.005948    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"]
Jul 31 15:01:24.006423 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.006229    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3793ccfc-bc3d-4858-80d4-c4a56b09a8fc" containerName="download-hf-model"
Jul 31 15:01:24.006423 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.006243    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3793ccfc-bc3d-4858-80d4-c4a56b09a8fc" containerName="download-hf-model"
Jul 31 15:01:24.006423 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.006250    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3793ccfc-bc3d-4858-80d4-c4a56b09a8fc" containerName="s3-init"
Jul 31 15:01:24.006423 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.006256    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3793ccfc-bc3d-4858-80d4-c4a56b09a8fc" containerName="s3-init"
Jul 31 15:01:24.006423 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.006319    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3793ccfc-bc3d-4858-80d4-c4a56b09a8fc" containerName="s3-init"
Jul 31 15:01:24.008219 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.008204    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.012234 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.012205    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-jn852\""
Jul 31 15:01:24.012419 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.012238    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 31 15:01:24.012419 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.012301    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 31 15:01:24.012419 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.012247    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 31 15:01:24.018402 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.018378    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"]
Jul 31 15:01:24.086452 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.086419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-dshm\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.086602 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.086469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsmm5\" (UniqueName: \"kubernetes.io/projected/272f1339-027a-4027-ba4f-db84afc646d3-kube-api-access-wsmm5\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.086602 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.086541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-home\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.086602 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.086578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.086602 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.086596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272f1339-027a-4027-ba4f-db84afc646d3-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.086733 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.086649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.187809 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.187770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsmm5\" (UniqueName: \"kubernetes.io/projected/272f1339-027a-4027-ba4f-db84afc646d3-kube-api-access-wsmm5\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.187978 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.187823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-home\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.187978 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.187941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.187978 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.187976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272f1339-027a-4027-ba4f-db84afc646d3-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.188134 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.188114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.188181 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.188164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-dshm\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.188215 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.188186    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-home\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.188303 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.188286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.188473 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.188455    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.190210 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.190183    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-dshm\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.190392 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.190375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272f1339-027a-4027-ba4f-db84afc646d3-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.196699 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.196675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsmm5\" (UniqueName: \"kubernetes.io/projected/272f1339-027a-4027-ba4f-db84afc646d3-kube-api-access-wsmm5\") pod \"fc-smoke-test-kserve-7f4687897-f6r4n\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.317949 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.317869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:24.450956 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:24.450928    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"]
Jul 31 15:01:24.452705 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:01:24.452680    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272f1339_027a_4027_ba4f_db84afc646d3.slice/crio-f00aea648097d1c4865963bc993437d6709ca77c27828b01752d3af9c718703b WatchSource:0}: Error finding container f00aea648097d1c4865963bc993437d6709ca77c27828b01752d3af9c718703b: Status 404 returned error can't find the container with id f00aea648097d1c4865963bc993437d6709ca77c27828b01752d3af9c718703b
Jul 31 15:01:25.358373 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:25.357338    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n" event={"ID":"272f1339-027a-4027-ba4f-db84afc646d3","Type":"ContainerStarted","Data":"f00aea648097d1c4865963bc993437d6709ca77c27828b01752d3af9c718703b"}
Jul 31 15:01:27.368967 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:27.368926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n" event={"ID":"272f1339-027a-4027-ba4f-db84afc646d3","Type":"ContainerStarted","Data":"925773920df030faf7be81d4118fb32e53fe1836312d1bd0e63bceaa18c8ddf9"}
Jul 31 15:01:27.390342 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:27.390238    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n" podStartSLOduration=1.696872091 podStartE2EDuration="4.390220229s" totalImagesPullingTime="2.693348138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:01:27.388901476 +0000 UTC m=+785.398113347" watchObservedRunningTime="2026-07-31 15:01:27.390220229 +0000 UTC m=+785.399432101"
Jul 31 15:01:28.318417 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.318384    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:28.638681 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.638595    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 31 15:01:28.641924 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.641902    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.645372 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.645334    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 31 15:01:28.645946 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.645927    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:01:28.648580 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.648563    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 31 15:01:28.648916 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.648899    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-tt4kb\""
Jul 31 15:01:28.649228 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.649212    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-czjml\""
Jul 31 15:01:28.649615 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.649596    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 31 15:01:28.650099 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.649944    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 31 15:01:28.660825 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.660800    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 31 15:01:28.739920 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.739881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jwsb\" (UniqueName: \"kubernetes.io/projected/53013629-4be2-4ce9-8ec3-6244e3467b9b-kube-api-access-7jwsb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.740200 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.740181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.740310 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.740295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.740426 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.740413    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.740534 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.740521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.740665 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.740653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.740755 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.740743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53013629-4be2-4ce9-8ec3-6244e3467b9b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.740854 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.740841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53013629-4be2-4ce9-8ec3-6244e3467b9b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.842003 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.841936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53013629-4be2-4ce9-8ec3-6244e3467b9b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.842225 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.842043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jwsb\" (UniqueName: \"kubernetes.io/projected/53013629-4be2-4ce9-8ec3-6244e3467b9b-kube-api-access-7jwsb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.842225 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.842073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.842225 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.842102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.842225 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.842127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.842225 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.842180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.842529 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.842256    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.842529 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.842286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53013629-4be2-4ce9-8ec3-6244e3467b9b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.842985 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.842955    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53013629-4be2-4ce9-8ec3-6244e3467b9b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.844068 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.844027    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.844322 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.844277    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.844573 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.844558    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.844810 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.844796    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.850897 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.850850    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.855756 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.855727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53013629-4be2-4ce9-8ec3-6244e3467b9b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.857390 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.857370    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jwsb\" (UniqueName: \"kubernetes.io/projected/53013629-4be2-4ce9-8ec3-6244e3467b9b-kube-api-access-7jwsb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:28.962659 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:28.962563    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:01:29.123473 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:29.123340    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 31 15:01:29.125887 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:01:29.125860    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53013629_4be2_4ce9_8ec3_6244e3467b9b.slice/crio-20476e3a9deccef468ad9a6ace7e4c9f54b29bab757b8320da1c86475282d741 WatchSource:0}: Error finding container 20476e3a9deccef468ad9a6ace7e4c9f54b29bab757b8320da1c86475282d741: Status 404 returned error can't find the container with id 20476e3a9deccef468ad9a6ace7e4c9f54b29bab757b8320da1c86475282d741
Jul 31 15:01:29.381394 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:29.381339    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"53013629-4be2-4ce9-8ec3-6244e3467b9b","Type":"ContainerStarted","Data":"f9192c620c058f787085cb511dd2f05209027f7005f1023e92e1db934a79a709"}
Jul 31 15:01:29.381588 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:29.381419    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"53013629-4be2-4ce9-8ec3-6244e3467b9b","Type":"ContainerStarted","Data":"20476e3a9deccef468ad9a6ace7e4c9f54b29bab757b8320da1c86475282d741"}
Jul 31 15:01:32.394584 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:32.394550    2578 generic.go:356] "Generic (PLEG): container finished" podID="53013629-4be2-4ce9-8ec3-6244e3467b9b" containerID="f9192c620c058f787085cb511dd2f05209027f7005f1023e92e1db934a79a709" exitCode=0
Jul 31 15:01:32.395095 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:32.394624    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"53013629-4be2-4ce9-8ec3-6244e3467b9b","Type":"ContainerDied","Data":"f9192c620c058f787085cb511dd2f05209027f7005f1023e92e1db934a79a709"}
Jul 31 15:01:34.327435 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:34.318189    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:34.345122 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:34.340258    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:34.425616 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:34.425381    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:01:35.415979 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:35.415899    2578 generic.go:356] "Generic (PLEG): container finished" podID="9020da08-566c-41f1-bf8b-339a5d8ed640" containerID="a605110fb5a49b2555d075edd8ea4cf0de128e27595e2576719e5b074627dbf3" exitCode=0
Jul 31 15:01:35.416441 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:35.416257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v" event={"ID":"9020da08-566c-41f1-bf8b-339a5d8ed640","Type":"ContainerDied","Data":"a605110fb5a49b2555d075edd8ea4cf0de128e27595e2576719e5b074627dbf3"}
Jul 31 15:01:35.416692 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:35.416668    2578 scope.go:117] "RemoveContainer" containerID="a605110fb5a49b2555d075edd8ea4cf0de128e27595e2576719e5b074627dbf3"
Jul 31 15:01:35.417824 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:35.417786    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:01:35.417999 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:35.417903    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:01:35.417999 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:35.417949    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:01:36.123558 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:36.123515    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 15:01:36.424662 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:36.424575    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v" event={"ID":"9020da08-566c-41f1-bf8b-339a5d8ed640","Type":"ContainerStarted","Data":"9bfb4fe8b56af8eaa85851de02537bb7a472cd79a9ca52e5143570294c850f7f"}
Jul 31 15:01:36.684673 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:36.684580    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 15:01:36.685603 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:36.685568    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.15:15021/healthz/ready\": dial tcp 10.134.0.15:15021: connect: connection refused" start-of-body=
Jul 31 15:01:36.685720 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:36.685629    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v" podUID="9020da08-566c-41f1-bf8b-339a5d8ed640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.15:15021/healthz/ready\": dial tcp 10.134.0.15:15021: connect: connection refused"
Jul 31 15:01:37.684740 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:37.684699    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.15:15021/healthz/ready\": dial tcp 10.134.0.15:15021: connect: connection refused" start-of-body=
Jul 31 15:01:37.685227 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:37.684763    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v" podUID="9020da08-566c-41f1-bf8b-339a5d8ed640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.15:15021/healthz/ready\": dial tcp 10.134.0.15:15021: connect: connection refused"
Jul 31 15:01:38.684564 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:38.684524    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.15:15021/healthz/ready\": dial tcp 10.134.0.15:15021: connect: connection refused" start-of-body=
Jul 31 15:01:38.684742 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:38.684589    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v" podUID="9020da08-566c-41f1-bf8b-339a5d8ed640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.15:15021/healthz/ready\": dial tcp 10.134.0.15:15021: connect: connection refused"
Jul 31 15:01:39.686009 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:39.685525    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.15:15021/healthz/ready\": dial tcp 10.134.0.15:15021: connect: connection refused" start-of-body=
Jul 31 15:01:39.686009 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:39.685610    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v" podUID="9020da08-566c-41f1-bf8b-339a5d8ed640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.15:15021/healthz/ready\": dial tcp 10.134.0.15:15021: connect: connection refused"
Jul 31 15:01:40.684911 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:40.684791    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.15:15021/healthz/ready\": dial tcp 10.134.0.15:15021: connect: connection refused" start-of-body=
Jul 31 15:01:40.684911 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:40.684871    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v" podUID="9020da08-566c-41f1-bf8b-339a5d8ed640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.15:15021/healthz/ready\": dial tcp 10.134.0.15:15021: connect: connection refused"
Jul 31 15:01:41.685685 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:41.685560    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.15:15021/healthz/ready\": dial tcp 10.134.0.15:15021: connect: connection refused" start-of-body=
Jul 31 15:01:41.685685 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:41.685623    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v" podUID="9020da08-566c-41f1-bf8b-339a5d8ed640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.15:15021/healthz/ready\": dial tcp 10.134.0.15:15021: connect: connection refused"
Jul 31 15:01:43.686258 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:43.685985    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 31 15:01:43.686258 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:43.686077    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v" podUID="9020da08-566c-41f1-bf8b-339a5d8ed640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 31 15:01:44.009019 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:44.008922    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 15:01:44.009779 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:44.009719    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 15:01:44.011101 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:01:44.011077    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-vqq7v"
Jul 31 15:02:05.583600 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:05.583559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"53013629-4be2-4ce9-8ec3-6244e3467b9b","Type":"ContainerStarted","Data":"8fb76bf1b6953101d590abc3a2524c234da8e5e625954940179f66e3cbbdcee2"}
Jul 31 15:02:05.608977 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:05.608919    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.782827787 podStartE2EDuration="37.608904661s" totalImagesPullingTime="32.30954725s" totalInitContainerRuntime="2.516529624s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:01:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:02:05.607625566 +0000 UTC m=+823.616837436" watchObservedRunningTime="2026-07-31 15:02:05.608904661 +0000 UTC m=+823.618116530"
Jul 31 15:02:06.452362 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:06.452316    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"]
Jul 31 15:02:06.463657 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:02:06.463153    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:02:06.463657 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:02:06.463657 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:02:06.463657 ip-10-0-133-32 kubenswrapper[2578]:  > containerID="925773920df030faf7be81d4118fb32e53fe1836312d1bd0e63bceaa18c8ddf9" cmd=["/bin/sleep","15"]
Jul 31 15:02:06.463657 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:06.463232    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n" message="" err=<
Jul 31 15:02:06.463657 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:02:06.463657 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:02:06.463657 ip-10-0-133-32 kubenswrapper[2578]:  >
Jul 31 15:02:06.463657 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:02:06.463250    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:02:06.463657 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:02:06.463657 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:02:06.463657 ip-10-0-133-32 kubenswrapper[2578]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n" podUID="272f1339-027a-4027-ba4f-db84afc646d3" containerName="main" containerID="cri-o://925773920df030faf7be81d4118fb32e53fe1836312d1bd0e63bceaa18c8ddf9"
Jul 31 15:02:06.463657 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:06.463301    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n" podUID="272f1339-027a-4027-ba4f-db84afc646d3" containerName="main" containerID="cri-o://925773920df030faf7be81d4118fb32e53fe1836312d1bd0e63bceaa18c8ddf9" gracePeriod=60
Jul 31 15:02:07.250804 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.250773    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:02:07.288975 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.288945    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsmm5\" (UniqueName: \"kubernetes.io/projected/272f1339-027a-4027-ba4f-db84afc646d3-kube-api-access-wsmm5\") pod \"272f1339-027a-4027-ba4f-db84afc646d3\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") "
Jul 31 15:02:07.289147 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.288990    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-model-cache\") pod \"272f1339-027a-4027-ba4f-db84afc646d3\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") "
Jul 31 15:02:07.289147 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.289016    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-dshm\") pod \"272f1339-027a-4027-ba4f-db84afc646d3\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") "
Jul 31 15:02:07.289147 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.289032    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-tmp-dir\") pod \"272f1339-027a-4027-ba4f-db84afc646d3\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") "
Jul 31 15:02:07.289322 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.289210    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-home\") pod \"272f1339-027a-4027-ba4f-db84afc646d3\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") "
Jul 31 15:02:07.289322 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.289278    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "272f1339-027a-4027-ba4f-db84afc646d3" (UID: "272f1339-027a-4027-ba4f-db84afc646d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:02:07.289322 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.289292    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272f1339-027a-4027-ba4f-db84afc646d3-tls-certs\") pod \"272f1339-027a-4027-ba4f-db84afc646d3\" (UID: \"272f1339-027a-4027-ba4f-db84afc646d3\") "
Jul 31 15:02:07.289322 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.289292    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "272f1339-027a-4027-ba4f-db84afc646d3" (UID: "272f1339-027a-4027-ba4f-db84afc646d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:02:07.289564 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.289486    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-home" (OuterVolumeSpecName: "home") pod "272f1339-027a-4027-ba4f-db84afc646d3" (UID: "272f1339-027a-4027-ba4f-db84afc646d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:02:07.289564 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.289541    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:02:07.289564 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.289551    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:02:07.289564 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.289561    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:02:07.291407 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.291335    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272f1339-027a-4027-ba4f-db84afc646d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "272f1339-027a-4027-ba4f-db84afc646d3" (UID: "272f1339-027a-4027-ba4f-db84afc646d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:02:07.291517 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.291405    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-dshm" (OuterVolumeSpecName: "dshm") pod "272f1339-027a-4027-ba4f-db84afc646d3" (UID: "272f1339-027a-4027-ba4f-db84afc646d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:02:07.291517 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.291455    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272f1339-027a-4027-ba4f-db84afc646d3-kube-api-access-wsmm5" (OuterVolumeSpecName: "kube-api-access-wsmm5") pod "272f1339-027a-4027-ba4f-db84afc646d3" (UID: "272f1339-027a-4027-ba4f-db84afc646d3"). InnerVolumeSpecName "kube-api-access-wsmm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:02:07.390395 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.390295    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272f1339-027a-4027-ba4f-db84afc646d3-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:02:07.390395 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.390338    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsmm5\" (UniqueName: \"kubernetes.io/projected/272f1339-027a-4027-ba4f-db84afc646d3-kube-api-access-wsmm5\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:02:07.390395 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.390383    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272f1339-027a-4027-ba4f-db84afc646d3-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:02:07.594431 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.594398    2578 generic.go:356] "Generic (PLEG): container finished" podID="272f1339-027a-4027-ba4f-db84afc646d3" containerID="925773920df030faf7be81d4118fb32e53fe1836312d1bd0e63bceaa18c8ddf9" exitCode=0
Jul 31 15:02:07.594603 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.594443    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"
Jul 31 15:02:07.594603 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.594447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n" event={"ID":"272f1339-027a-4027-ba4f-db84afc646d3","Type":"ContainerDied","Data":"925773920df030faf7be81d4118fb32e53fe1836312d1bd0e63bceaa18c8ddf9"}
Jul 31 15:02:07.594603 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.594584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n" event={"ID":"272f1339-027a-4027-ba4f-db84afc646d3","Type":"ContainerDied","Data":"f00aea648097d1c4865963bc993437d6709ca77c27828b01752d3af9c718703b"}
Jul 31 15:02:07.594745 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.594622    2578 scope.go:117] "RemoveContainer" containerID="925773920df030faf7be81d4118fb32e53fe1836312d1bd0e63bceaa18c8ddf9"
Jul 31 15:02:07.613842 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.613815    2578 scope.go:117] "RemoveContainer" containerID="925773920df030faf7be81d4118fb32e53fe1836312d1bd0e63bceaa18c8ddf9"
Jul 31 15:02:07.614131 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:02:07.614108    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925773920df030faf7be81d4118fb32e53fe1836312d1bd0e63bceaa18c8ddf9\": container with ID starting with 925773920df030faf7be81d4118fb32e53fe1836312d1bd0e63bceaa18c8ddf9 not found: ID does not exist" containerID="925773920df030faf7be81d4118fb32e53fe1836312d1bd0e63bceaa18c8ddf9"
Jul 31 15:02:07.614217 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.614140    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925773920df030faf7be81d4118fb32e53fe1836312d1bd0e63bceaa18c8ddf9"} err="failed to get container status \"925773920df030faf7be81d4118fb32e53fe1836312d1bd0e63bceaa18c8ddf9\": rpc error: code = NotFound desc = could not find container \"925773920df030faf7be81d4118fb32e53fe1836312d1bd0e63bceaa18c8ddf9\": container with ID starting with 925773920df030faf7be81d4118fb32e53fe1836312d1bd0e63bceaa18c8ddf9 not found: ID does not exist"
Jul 31 15:02:07.626123 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.626097    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"]
Jul 31 15:02:07.630305 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:07.630269    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-f6r4n"]
Jul 31 15:02:08.489925 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:08.489896    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="272f1339-027a-4027-ba4f-db84afc646d3" path="/var/lib/kubelet/pods/272f1339-027a-4027-ba4f-db84afc646d3/volumes"
Jul 31 15:02:31.756765 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:31.756733    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"]
Jul 31 15:02:31.757257 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:31.757086    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272f1339-027a-4027-ba4f-db84afc646d3" containerName="main"
Jul 31 15:02:31.757257 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:31.757102    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="272f1339-027a-4027-ba4f-db84afc646d3" containerName="main"
Jul 31 15:02:31.757257 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:31.757163    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="272f1339-027a-4027-ba4f-db84afc646d3" containerName="main"
Jul 31 15:02:31.760118 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:31.760096    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"
Jul 31 15:02:31.765054 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:31.765015    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-lq9fn\""
Jul 31 15:02:31.765054 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:31.765039    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Jul 31 15:02:31.765244 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:31.765022    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Jul 31 15:02:31.765244 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:31.765044    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Jul 31 15:02:31.765244 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:31.765023    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-fwhrc\""
Jul 31 15:02:31.772092 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:31.772069    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"]
Jul 31 15:02:31.883494 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:31.883458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/403050f8-097c-41c3-b900-ba4d80559244-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558\" (UID: \"403050f8-097c-41c3-b900-ba4d80559244\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"
Jul 31 15:02:31.883664 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:31.883523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qtrc\" (UniqueName: \"kubernetes.io/projected/403050f8-097c-41c3-b900-ba4d80559244-kube-api-access-4qtrc\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558\" (UID: \"403050f8-097c-41c3-b900-ba4d80559244\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"
Jul 31 15:02:31.984573 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:31.984535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/403050f8-097c-41c3-b900-ba4d80559244-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558\" (UID: \"403050f8-097c-41c3-b900-ba4d80559244\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"
Jul 31 15:02:31.984771 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:31.984645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qtrc\" (UniqueName: \"kubernetes.io/projected/403050f8-097c-41c3-b900-ba4d80559244-kube-api-access-4qtrc\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558\" (UID: \"403050f8-097c-41c3-b900-ba4d80559244\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"
Jul 31 15:02:31.987043 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:31.987013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/403050f8-097c-41c3-b900-ba4d80559244-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558\" (UID: \"403050f8-097c-41c3-b900-ba4d80559244\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"
Jul 31 15:02:31.995811 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:31.995782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qtrc\" (UniqueName: \"kubernetes.io/projected/403050f8-097c-41c3-b900-ba4d80559244-kube-api-access-4qtrc\") pod \"fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558\" (UID: \"403050f8-097c-41c3-b900-ba4d80559244\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"
Jul 31 15:02:32.071531 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:32.071442    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"
Jul 31 15:02:32.210060 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:32.210035    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"]
Jul 31 15:02:32.212815 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:02:32.212788    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403050f8_097c_41c3_b900_ba4d80559244.slice/crio-01b8e558b845c37fcc1f3eeed01fbf24a6cd9271cfa99232afcd1d673fb8cc37 WatchSource:0}: Error finding container 01b8e558b845c37fcc1f3eeed01fbf24a6cd9271cfa99232afcd1d673fb8cc37: Status 404 returned error can't find the container with id 01b8e558b845c37fcc1f3eeed01fbf24a6cd9271cfa99232afcd1d673fb8cc37
Jul 31 15:02:32.695870 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:32.695825    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558" event={"ID":"403050f8-097c-41c3-b900-ba4d80559244","Type":"ContainerStarted","Data":"01b8e558b845c37fcc1f3eeed01fbf24a6cd9271cfa99232afcd1d673fb8cc37"}
Jul 31 15:02:35.712500 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:35.712460    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558" event={"ID":"403050f8-097c-41c3-b900-ba4d80559244","Type":"ContainerStarted","Data":"c57166093828cd0bc69754de19789e69c23dc2904519b651b5605976d95c9e4f"}
Jul 31 15:02:35.712967 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:35.712606    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"
Jul 31 15:02:35.730327 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:02:35.730281    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558" podStartSLOduration=2.183943893 podStartE2EDuration="4.730264514s" totalImagesPullingTime="2.546320621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:02:35.72865166 +0000 UTC m=+853.737863529" watchObservedRunningTime="2026-07-31 15:02:35.730264514 +0000 UTC m=+853.739476383"
Jul 31 15:03:06.718658 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:06.718627    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"
Jul 31 15:03:13.055254 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:13.055216    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"]
Jul 31 15:03:16.718792 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:16.718698    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558" podUID="403050f8-097c-41c3-b900-ba4d80559244" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 15:03:26.718182 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:26.718132    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558" podUID="403050f8-097c-41c3-b900-ba4d80559244" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 15:03:28.074408 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:28.074369    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558" podUID="403050f8-097c-41c3-b900-ba4d80559244" containerName="main" containerID="cri-o://c57166093828cd0bc69754de19789e69c23dc2904519b651b5605976d95c9e4f" gracePeriod=45
Jul 31 15:03:28.312852 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:28.312828    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"
Jul 31 15:03:28.468387 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:28.468282    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/403050f8-097c-41c3-b900-ba4d80559244-tls-certs\") pod \"403050f8-097c-41c3-b900-ba4d80559244\" (UID: \"403050f8-097c-41c3-b900-ba4d80559244\") "
Jul 31 15:03:28.468387 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:28.468339    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qtrc\" (UniqueName: \"kubernetes.io/projected/403050f8-097c-41c3-b900-ba4d80559244-kube-api-access-4qtrc\") pod \"403050f8-097c-41c3-b900-ba4d80559244\" (UID: \"403050f8-097c-41c3-b900-ba4d80559244\") "
Jul 31 15:03:28.470453 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:28.470420    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403050f8-097c-41c3-b900-ba4d80559244-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "403050f8-097c-41c3-b900-ba4d80559244" (UID: "403050f8-097c-41c3-b900-ba4d80559244"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:03:28.470567 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:28.470540    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403050f8-097c-41c3-b900-ba4d80559244-kube-api-access-4qtrc" (OuterVolumeSpecName: "kube-api-access-4qtrc") pod "403050f8-097c-41c3-b900-ba4d80559244" (UID: "403050f8-097c-41c3-b900-ba4d80559244"). InnerVolumeSpecName "kube-api-access-4qtrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:03:28.569229 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:28.569183    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/403050f8-097c-41c3-b900-ba4d80559244-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:03:28.569229 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:28.569227    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qtrc\" (UniqueName: \"kubernetes.io/projected/403050f8-097c-41c3-b900-ba4d80559244-kube-api-access-4qtrc\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:03:28.924624 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:28.924593    2578 generic.go:356] "Generic (PLEG): container finished" podID="403050f8-097c-41c3-b900-ba4d80559244" containerID="c57166093828cd0bc69754de19789e69c23dc2904519b651b5605976d95c9e4f" exitCode=0
Jul 31 15:03:28.924779 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:28.924662    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"
Jul 31 15:03:28.924779 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:28.924681    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558" event={"ID":"403050f8-097c-41c3-b900-ba4d80559244","Type":"ContainerDied","Data":"c57166093828cd0bc69754de19789e69c23dc2904519b651b5605976d95c9e4f"}
Jul 31 15:03:28.924779 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:28.924738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558" event={"ID":"403050f8-097c-41c3-b900-ba4d80559244","Type":"ContainerDied","Data":"01b8e558b845c37fcc1f3eeed01fbf24a6cd9271cfa99232afcd1d673fb8cc37"}
Jul 31 15:03:28.924779 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:28.924761    2578 scope.go:117] "RemoveContainer" containerID="c57166093828cd0bc69754de19789e69c23dc2904519b651b5605976d95c9e4f"
Jul 31 15:03:28.934505 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:28.934483    2578 scope.go:117] "RemoveContainer" containerID="c57166093828cd0bc69754de19789e69c23dc2904519b651b5605976d95c9e4f"
Jul 31 15:03:28.934781 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:03:28.934762    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57166093828cd0bc69754de19789e69c23dc2904519b651b5605976d95c9e4f\": container with ID starting with c57166093828cd0bc69754de19789e69c23dc2904519b651b5605976d95c9e4f not found: ID does not exist" containerID="c57166093828cd0bc69754de19789e69c23dc2904519b651b5605976d95c9e4f"
Jul 31 15:03:28.934838 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:28.934789    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57166093828cd0bc69754de19789e69c23dc2904519b651b5605976d95c9e4f"} err="failed to get container status \"c57166093828cd0bc69754de19789e69c23dc2904519b651b5605976d95c9e4f\": rpc error: code = NotFound desc = could not find container \"c57166093828cd0bc69754de19789e69c23dc2904519b651b5605976d95c9e4f\": container with ID starting with c57166093828cd0bc69754de19789e69c23dc2904519b651b5605976d95c9e4f not found: ID does not exist"
Jul 31 15:03:28.943042 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:28.943019    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"]
Jul 31 15:03:28.946715 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:28.946696    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-69bbd6f4df-vf558"]
Jul 31 15:03:30.491529 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:30.491492    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="403050f8-097c-41c3-b900-ba4d80559244" path="/var/lib/kubelet/pods/403050f8-097c-41c3-b900-ba4d80559244/volumes"
Jul 31 15:03:33.329618 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.329594    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"]
Jul 31 15:03:33.329927 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.329863    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="403050f8-097c-41c3-b900-ba4d80559244" containerName="main"
Jul 31 15:03:33.329927 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.329873    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="403050f8-097c-41c3-b900-ba4d80559244" containerName="main"
Jul 31 15:03:33.330003 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.329929    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="403050f8-097c-41c3-b900-ba4d80559244" containerName="main"
Jul 31 15:03:33.335676 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.335659    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.338761 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.338738    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-4nkg9\""
Jul 31 15:03:33.339379 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.339342    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Jul 31 15:03:33.339453 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.339387    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Jul 31 15:03:33.339844 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.339829    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Jul 31 15:03:33.348918 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.348895    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"]
Jul 31 15:03:33.510937 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.510892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00e6012-747f-4673-8e01-4cec0683b330-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.511137 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.510958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00e6012-747f-4673-8e01-4cec0683b330-istio-podinfo\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.511137 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.511000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-workload-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.511137 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.511038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-istio-data\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.511137 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.511085    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00e6012-747f-4673-8e01-4cec0683b330-istio-token\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.511137 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.511125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-credential-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.511325 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.511149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-istio-envoy\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.511325 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.511205    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-workload-certs\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.511325 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.511232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ckcw\" (UniqueName: \"kubernetes.io/projected/d00e6012-747f-4673-8e01-4cec0683b330-kube-api-access-6ckcw\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.612224 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.612135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-workload-certs\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.612224 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.612180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ckcw\" (UniqueName: \"kubernetes.io/projected/d00e6012-747f-4673-8e01-4cec0683b330-kube-api-access-6ckcw\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.612469 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.612279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00e6012-747f-4673-8e01-4cec0683b330-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.612469 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.612299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00e6012-747f-4673-8e01-4cec0683b330-istio-podinfo\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.612469 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.612329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-workload-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.612469 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.612371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-istio-data\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.612469 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.612398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00e6012-747f-4673-8e01-4cec0683b330-istio-token\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.612715 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.612620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-workload-certs\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.612715 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.612684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-credential-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.612819 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.612720    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-istio-envoy\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.612819 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.612799    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-workload-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.612929 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.612889    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-istio-data\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.613079 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.613055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00e6012-747f-4673-8e01-4cec0683b330-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.613142 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.613118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-credential-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.614790 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.614767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-istio-envoy\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.614884 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.614813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00e6012-747f-4673-8e01-4cec0683b330-istio-podinfo\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.627181 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.627155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00e6012-747f-4673-8e01-4cec0683b330-istio-token\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.628008 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.627986    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ckcw\" (UniqueName: \"kubernetes.io/projected/d00e6012-747f-4673-8e01-4cec0683b330-kube-api-access-6ckcw\") pod \"router-gateway-1-openshift-default-5944c46dd4-86bpx\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.645735 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.645703    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:33.787487 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.787456    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"]
Jul 31 15:03:33.789576 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:03:33.789544    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00e6012_747f_4673_8e01_4cec0683b330.slice/crio-8a40bd9bf0f78765ba4ee5bee3859d400d747cc416e1bb474fb9e6544da6a85a WatchSource:0}: Error finding container 8a40bd9bf0f78765ba4ee5bee3859d400d747cc416e1bb474fb9e6544da6a85a: Status 404 returned error can't find the container with id 8a40bd9bf0f78765ba4ee5bee3859d400d747cc416e1bb474fb9e6544da6a85a
Jul 31 15:03:33.792076 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.792011    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:03:33.792184 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.792111    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:03:33.792230 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.792190    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:03:33.948302 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.948219    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx" event={"ID":"d00e6012-747f-4673-8e01-4cec0683b330","Type":"ContainerStarted","Data":"b595119329ebf63c1eaab1f7c3e952ae785aabe7c946f5a8bcae7775057497bd"}
Jul 31 15:03:33.948302 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.948280    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx" event={"ID":"d00e6012-747f-4673-8e01-4cec0683b330","Type":"ContainerStarted","Data":"8a40bd9bf0f78765ba4ee5bee3859d400d747cc416e1bb474fb9e6544da6a85a"}
Jul 31 15:03:33.970199 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:33.970137    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx" podStartSLOduration=0.970113895 podStartE2EDuration="970.113895ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:03:33.968714649 +0000 UTC m=+911.977926543" watchObservedRunningTime="2026-07-31 15:03:33.970113895 +0000 UTC m=+911.979325769"
Jul 31 15:03:34.646450 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:34.645994    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:34.647491 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:34.647451    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx" podUID="d00e6012-747f-4673-8e01-4cec0683b330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused"
Jul 31 15:03:35.650100 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:35.650068    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:35.959124 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:35.959031    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:35.959943 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:35.959924    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:03:55.425237 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.425202    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"]
Jul 31 15:03:55.428257 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.428234    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.431916 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.431436    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Jul 31 15:03:55.431916 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.431436    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Jul 31 15:03:55.431916 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.431441    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-429pk\""
Jul 31 15:03:55.450163 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.450132    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"]
Jul 31 15:03:55.450481 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.450463    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Jul 31 15:03:55.483942 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.483902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-workload-certs\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.484118 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.483960    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-istio-data\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.484118 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.484038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-credential-socket\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.484118 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.484090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06911a79-381d-4b54-8699-4d0c13f9384a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.484285 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.484124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06911a79-381d-4b54-8699-4d0c13f9384a-istio-podinfo\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.484285 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.484165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-istio-envoy\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.484285 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.484212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06911a79-381d-4b54-8699-4d0c13f9384a-istio-token\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.484285 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.484247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-workload-socket\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.484285 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.484273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj8sw\" (UniqueName: \"kubernetes.io/projected/06911a79-381d-4b54-8699-4d0c13f9384a-kube-api-access-qj8sw\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.585121 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.585081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-istio-data\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.585121 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.585125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-credential-socket\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.585408 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.585145    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06911a79-381d-4b54-8699-4d0c13f9384a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.585408 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.585183    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06911a79-381d-4b54-8699-4d0c13f9384a-istio-podinfo\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.585408 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.585212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-istio-envoy\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.585408 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.585231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06911a79-381d-4b54-8699-4d0c13f9384a-istio-token\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.585408 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.585266    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-workload-socket\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.585660 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.585403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj8sw\" (UniqueName: \"kubernetes.io/projected/06911a79-381d-4b54-8699-4d0c13f9384a-kube-api-access-qj8sw\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.585660 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.585500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-workload-certs\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.585660 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.585601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-credential-socket\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.585660 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.585634    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-istio-data\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.585880 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.585855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-workload-certs\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.585971 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.585950    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06911a79-381d-4b54-8699-4d0c13f9384a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.586030 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.585959    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-workload-socket\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.587531 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.587510    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-istio-envoy\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.587661 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.587645    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06911a79-381d-4b54-8699-4d0c13f9384a-istio-podinfo\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.594752 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.594730    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06911a79-381d-4b54-8699-4d0c13f9384a-istio-token\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.594935 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.594911    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj8sw\" (UniqueName: \"kubernetes.io/projected/06911a79-381d-4b54-8699-4d0c13f9384a-kube-api-access-qj8sw\") pod \"router-gateway-1-openshift-default-78668f9df6-txht6\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.764887 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.764847    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:55.923467 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.923441    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"]
Jul 31 15:03:55.925678 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:03:55.925638    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06911a79_381d_4b54_8699_4d0c13f9384a.slice/crio-699949263b990616521dd02ff15772e1f35c272da11a9feb08c9977ab9f50103 WatchSource:0}: Error finding container 699949263b990616521dd02ff15772e1f35c272da11a9feb08c9977ab9f50103: Status 404 returned error can't find the container with id 699949263b990616521dd02ff15772e1f35c272da11a9feb08c9977ab9f50103
Jul 31 15:03:55.927750 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.927715    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:03:55.927821 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.927795    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:03:55.927862 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:55.927834    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:03:56.065566 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:56.065460    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6" event={"ID":"06911a79-381d-4b54-8699-4d0c13f9384a","Type":"ContainerStarted","Data":"610053d59c675650acda2603086678542efefef19bde41994d15af07210377c5"}
Jul 31 15:03:56.065743 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:56.065589    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6" event={"ID":"06911a79-381d-4b54-8699-4d0c13f9384a","Type":"ContainerStarted","Data":"699949263b990616521dd02ff15772e1f35c272da11a9feb08c9977ab9f50103"}
Jul 31 15:03:56.087594 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:56.087468    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6" podStartSLOduration=1.0874506290000001 podStartE2EDuration="1.087450629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:03:56.086915806 +0000 UTC m=+934.096127706" watchObservedRunningTime="2026-07-31 15:03:56.087450629 +0000 UTC m=+934.096662507"
Jul 31 15:03:56.765751 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:56.765712    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:56.770834 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:56.770806    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:57.070014 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:57.069928    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:03:57.070990 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:03:57.070967    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:04:00.121625 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:00.121544    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"]
Jul 31 15:04:00.122019 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:00.121745    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx" podUID="d00e6012-747f-4673-8e01-4cec0683b330" containerName="istio-proxy" containerID="cri-o://b595119329ebf63c1eaab1f7c3e952ae785aabe7c946f5a8bcae7775057497bd" gracePeriod=30
Jul 31 15:04:02.966362 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:02.966328    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:04:03.050094 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.050013    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00e6012-747f-4673-8e01-4cec0683b330-istio-podinfo\") pod \"d00e6012-747f-4673-8e01-4cec0683b330\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") "
Jul 31 15:04:03.050094 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.050057    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-workload-certs\") pod \"d00e6012-747f-4673-8e01-4cec0683b330\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") "
Jul 31 15:04:03.050302 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.050109    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ckcw\" (UniqueName: \"kubernetes.io/projected/d00e6012-747f-4673-8e01-4cec0683b330-kube-api-access-6ckcw\") pod \"d00e6012-747f-4673-8e01-4cec0683b330\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") "
Jul 31 15:04:03.050302 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.050139    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-istio-data\") pod \"d00e6012-747f-4673-8e01-4cec0683b330\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") "
Jul 31 15:04:03.050302 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.050165    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-workload-socket\") pod \"d00e6012-747f-4673-8e01-4cec0683b330\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") "
Jul 31 15:04:03.050302 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.050193    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-credential-socket\") pod \"d00e6012-747f-4673-8e01-4cec0683b330\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") "
Jul 31 15:04:03.050302 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.050261    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00e6012-747f-4673-8e01-4cec0683b330-istio-token\") pod \"d00e6012-747f-4673-8e01-4cec0683b330\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") "
Jul 31 15:04:03.050577 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.050307    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00e6012-747f-4673-8e01-4cec0683b330-istiod-ca-cert\") pod \"d00e6012-747f-4673-8e01-4cec0683b330\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") "
Jul 31 15:04:03.050577 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.050335    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-istio-envoy\") pod \"d00e6012-747f-4673-8e01-4cec0683b330\" (UID: \"d00e6012-747f-4673-8e01-4cec0683b330\") "
Jul 31 15:04:03.050577 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.050452    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-istio-data" (OuterVolumeSpecName: "istio-data") pod "d00e6012-747f-4673-8e01-4cec0683b330" (UID: "d00e6012-747f-4673-8e01-4cec0683b330"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:03.050577 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.050467    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d00e6012-747f-4673-8e01-4cec0683b330" (UID: "d00e6012-747f-4673-8e01-4cec0683b330"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:03.050769 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.050637    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d00e6012-747f-4673-8e01-4cec0683b330" (UID: "d00e6012-747f-4673-8e01-4cec0683b330"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:03.050769 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.050705    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00e6012-747f-4673-8e01-4cec0683b330-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d00e6012-747f-4673-8e01-4cec0683b330" (UID: "d00e6012-747f-4673-8e01-4cec0683b330"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:04:03.050862 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.050781    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d00e6012-747f-4673-8e01-4cec0683b330" (UID: "d00e6012-747f-4673-8e01-4cec0683b330"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:03.051029 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.050971    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:03.051029 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.050998    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:03.051029 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.051013    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00e6012-747f-4673-8e01-4cec0683b330-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:03.051029 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.051025    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:03.051247 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.051037    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:03.052555 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.052522    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00e6012-747f-4673-8e01-4cec0683b330-istio-token" (OuterVolumeSpecName: "istio-token") pod "d00e6012-747f-4673-8e01-4cec0683b330" (UID: "d00e6012-747f-4673-8e01-4cec0683b330"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:04:03.052646 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.052550    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d00e6012-747f-4673-8e01-4cec0683b330-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d00e6012-747f-4673-8e01-4cec0683b330" (UID: "d00e6012-747f-4673-8e01-4cec0683b330"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 15:04:03.052646 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.052616    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d00e6012-747f-4673-8e01-4cec0683b330" (UID: "d00e6012-747f-4673-8e01-4cec0683b330"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:03.052794 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.052774    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00e6012-747f-4673-8e01-4cec0683b330-kube-api-access-6ckcw" (OuterVolumeSpecName: "kube-api-access-6ckcw") pod "d00e6012-747f-4673-8e01-4cec0683b330" (UID: "d00e6012-747f-4673-8e01-4cec0683b330"). InnerVolumeSpecName "kube-api-access-6ckcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:04:03.096454 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.096420    2578 generic.go:356] "Generic (PLEG): container finished" podID="d00e6012-747f-4673-8e01-4cec0683b330" containerID="b595119329ebf63c1eaab1f7c3e952ae785aabe7c946f5a8bcae7775057497bd" exitCode=0
Jul 31 15:04:03.096591 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.096463    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx" event={"ID":"d00e6012-747f-4673-8e01-4cec0683b330","Type":"ContainerDied","Data":"b595119329ebf63c1eaab1f7c3e952ae785aabe7c946f5a8bcae7775057497bd"}
Jul 31 15:04:03.096591 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.096471    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"
Jul 31 15:04:03.096591 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.096547    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx" event={"ID":"d00e6012-747f-4673-8e01-4cec0683b330","Type":"ContainerDied","Data":"8a40bd9bf0f78765ba4ee5bee3859d400d747cc416e1bb474fb9e6544da6a85a"}
Jul 31 15:04:03.096591 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.096578    2578 scope.go:117] "RemoveContainer" containerID="b595119329ebf63c1eaab1f7c3e952ae785aabe7c946f5a8bcae7775057497bd"
Jul 31 15:04:03.104809 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.104793    2578 scope.go:117] "RemoveContainer" containerID="b595119329ebf63c1eaab1f7c3e952ae785aabe7c946f5a8bcae7775057497bd"
Jul 31 15:04:03.105047 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:04:03.105029    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b595119329ebf63c1eaab1f7c3e952ae785aabe7c946f5a8bcae7775057497bd\": container with ID starting with b595119329ebf63c1eaab1f7c3e952ae785aabe7c946f5a8bcae7775057497bd not found: ID does not exist" containerID="b595119329ebf63c1eaab1f7c3e952ae785aabe7c946f5a8bcae7775057497bd"
Jul 31 15:04:03.105105 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.105054    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b595119329ebf63c1eaab1f7c3e952ae785aabe7c946f5a8bcae7775057497bd"} err="failed to get container status \"b595119329ebf63c1eaab1f7c3e952ae785aabe7c946f5a8bcae7775057497bd\": rpc error: code = NotFound desc = could not find container \"b595119329ebf63c1eaab1f7c3e952ae785aabe7c946f5a8bcae7775057497bd\": container with ID starting with b595119329ebf63c1eaab1f7c3e952ae785aabe7c946f5a8bcae7775057497bd not found: ID does not exist"
Jul 31 15:04:03.121063 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.121034    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"]
Jul 31 15:04:03.126997 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.126972    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-86bpx"]
Jul 31 15:04:03.151678 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.151646    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00e6012-747f-4673-8e01-4cec0683b330-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:03.151678 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.151673    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00e6012-747f-4673-8e01-4cec0683b330-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:03.151678 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.151681    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00e6012-747f-4673-8e01-4cec0683b330-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:03.151866 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:03.151690    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ckcw\" (UniqueName: \"kubernetes.io/projected/d00e6012-747f-4673-8e01-4cec0683b330-kube-api-access-6ckcw\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:04.489103 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:04.489071    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d00e6012-747f-4673-8e01-4cec0683b330" path="/var/lib/kubelet/pods/d00e6012-747f-4673-8e01-4cec0683b330/volumes"
Jul 31 15:04:15.115614 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:15.115578    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"]
Jul 31 15:04:15.116184 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:15.115774    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6" podUID="06911a79-381d-4b54-8699-4d0c13f9384a" containerName="istio-proxy" containerID="cri-o://610053d59c675650acda2603086678542efefef19bde41994d15af07210377c5" gracePeriod=30
Jul 31 15:04:18.356359 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.356318    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:04:18.475901 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.475871    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-istio-envoy\") pod \"06911a79-381d-4b54-8699-4d0c13f9384a\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") "
Jul 31 15:04:18.476043 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.475920    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-istio-data\") pod \"06911a79-381d-4b54-8699-4d0c13f9384a\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") "
Jul 31 15:04:18.476043 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.475948    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06911a79-381d-4b54-8699-4d0c13f9384a-istio-token\") pod \"06911a79-381d-4b54-8699-4d0c13f9384a\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") "
Jul 31 15:04:18.476043 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.475968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06911a79-381d-4b54-8699-4d0c13f9384a-istio-podinfo\") pod \"06911a79-381d-4b54-8699-4d0c13f9384a\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") "
Jul 31 15:04:18.476043 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.475990    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj8sw\" (UniqueName: \"kubernetes.io/projected/06911a79-381d-4b54-8699-4d0c13f9384a-kube-api-access-qj8sw\") pod \"06911a79-381d-4b54-8699-4d0c13f9384a\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") "
Jul 31 15:04:18.476043 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.476014    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-workload-certs\") pod \"06911a79-381d-4b54-8699-4d0c13f9384a\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") "
Jul 31 15:04:18.476276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.476057    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-credential-socket\") pod \"06911a79-381d-4b54-8699-4d0c13f9384a\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") "
Jul 31 15:04:18.476276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.476110    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-workload-socket\") pod \"06911a79-381d-4b54-8699-4d0c13f9384a\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") "
Jul 31 15:04:18.476276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.476142    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06911a79-381d-4b54-8699-4d0c13f9384a-istiod-ca-cert\") pod \"06911a79-381d-4b54-8699-4d0c13f9384a\" (UID: \"06911a79-381d-4b54-8699-4d0c13f9384a\") "
Jul 31 15:04:18.476460 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.476436    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "06911a79-381d-4b54-8699-4d0c13f9384a" (UID: "06911a79-381d-4b54-8699-4d0c13f9384a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:18.476516 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.476468    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "06911a79-381d-4b54-8699-4d0c13f9384a" (UID: "06911a79-381d-4b54-8699-4d0c13f9384a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:18.476725 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.476658    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06911a79-381d-4b54-8699-4d0c13f9384a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "06911a79-381d-4b54-8699-4d0c13f9384a" (UID: "06911a79-381d-4b54-8699-4d0c13f9384a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:04:18.476725 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.476661    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "06911a79-381d-4b54-8699-4d0c13f9384a" (UID: "06911a79-381d-4b54-8699-4d0c13f9384a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:18.476900 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.476866    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-istio-data" (OuterVolumeSpecName: "istio-data") pod "06911a79-381d-4b54-8699-4d0c13f9384a" (UID: "06911a79-381d-4b54-8699-4d0c13f9384a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:18.478317 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.478296    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "06911a79-381d-4b54-8699-4d0c13f9384a" (UID: "06911a79-381d-4b54-8699-4d0c13f9384a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:18.478417 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.478311    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06911a79-381d-4b54-8699-4d0c13f9384a-istio-token" (OuterVolumeSpecName: "istio-token") pod "06911a79-381d-4b54-8699-4d0c13f9384a" (UID: "06911a79-381d-4b54-8699-4d0c13f9384a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:04:18.478518 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.478499    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06911a79-381d-4b54-8699-4d0c13f9384a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "06911a79-381d-4b54-8699-4d0c13f9384a" (UID: "06911a79-381d-4b54-8699-4d0c13f9384a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 15:04:18.478556 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.478509    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06911a79-381d-4b54-8699-4d0c13f9384a-kube-api-access-qj8sw" (OuterVolumeSpecName: "kube-api-access-qj8sw") pod "06911a79-381d-4b54-8699-4d0c13f9384a" (UID: "06911a79-381d-4b54-8699-4d0c13f9384a"). InnerVolumeSpecName "kube-api-access-qj8sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:04:18.577383 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.577333    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:18.577383 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.577378    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06911a79-381d-4b54-8699-4d0c13f9384a-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:18.577383 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.577387    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:18.577602 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.577396    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:18.577602 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.577404    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06911a79-381d-4b54-8699-4d0c13f9384a-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:18.577602 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.577412    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06911a79-381d-4b54-8699-4d0c13f9384a-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:18.577602 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.577420    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj8sw\" (UniqueName: \"kubernetes.io/projected/06911a79-381d-4b54-8699-4d0c13f9384a-kube-api-access-qj8sw\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:18.577602 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.577429    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:18.577602 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:18.577437    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06911a79-381d-4b54-8699-4d0c13f9384a-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:19.178365 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:19.178319    2578 generic.go:356] "Generic (PLEG): container finished" podID="06911a79-381d-4b54-8699-4d0c13f9384a" containerID="610053d59c675650acda2603086678542efefef19bde41994d15af07210377c5" exitCode=0
Jul 31 15:04:19.178534 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:19.178375    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"
Jul 31 15:04:19.178534 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:19.178389    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6" event={"ID":"06911a79-381d-4b54-8699-4d0c13f9384a","Type":"ContainerDied","Data":"610053d59c675650acda2603086678542efefef19bde41994d15af07210377c5"}
Jul 31 15:04:19.178534 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:19.178441    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6" event={"ID":"06911a79-381d-4b54-8699-4d0c13f9384a","Type":"ContainerDied","Data":"699949263b990616521dd02ff15772e1f35c272da11a9feb08c9977ab9f50103"}
Jul 31 15:04:19.178534 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:19.178470    2578 scope.go:117] "RemoveContainer" containerID="610053d59c675650acda2603086678542efefef19bde41994d15af07210377c5"
Jul 31 15:04:19.186696 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:19.186681    2578 scope.go:117] "RemoveContainer" containerID="610053d59c675650acda2603086678542efefef19bde41994d15af07210377c5"
Jul 31 15:04:19.186927 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:04:19.186907    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"610053d59c675650acda2603086678542efefef19bde41994d15af07210377c5\": container with ID starting with 610053d59c675650acda2603086678542efefef19bde41994d15af07210377c5 not found: ID does not exist" containerID="610053d59c675650acda2603086678542efefef19bde41994d15af07210377c5"
Jul 31 15:04:19.186991 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:19.186938    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"610053d59c675650acda2603086678542efefef19bde41994d15af07210377c5"} err="failed to get container status \"610053d59c675650acda2603086678542efefef19bde41994d15af07210377c5\": rpc error: code = NotFound desc = could not find container \"610053d59c675650acda2603086678542efefef19bde41994d15af07210377c5\": container with ID starting with 610053d59c675650acda2603086678542efefef19bde41994d15af07210377c5 not found: ID does not exist"
Jul 31 15:04:19.197841 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:19.197816    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"]
Jul 31 15:04:19.201455 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:19.201433    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-txht6"]
Jul 31 15:04:20.489761 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.489722    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06911a79-381d-4b54-8699-4d0c13f9384a" path="/var/lib/kubelet/pods/06911a79-381d-4b54-8699-4d0c13f9384a/volumes"
Jul 31 15:04:20.523051 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.523017    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"]
Jul 31 15:04:20.523313 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.523301    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06911a79-381d-4b54-8699-4d0c13f9384a" containerName="istio-proxy"
Jul 31 15:04:20.523382 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.523315    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="06911a79-381d-4b54-8699-4d0c13f9384a" containerName="istio-proxy"
Jul 31 15:04:20.523382 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.523334    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00e6012-747f-4673-8e01-4cec0683b330" containerName="istio-proxy"
Jul 31 15:04:20.523382 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.523339    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00e6012-747f-4673-8e01-4cec0683b330" containerName="istio-proxy"
Jul 31 15:04:20.523493 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.523403    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d00e6012-747f-4673-8e01-4cec0683b330" containerName="istio-proxy"
Jul 31 15:04:20.523493 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.523410    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="06911a79-381d-4b54-8699-4d0c13f9384a" containerName="istio-proxy"
Jul 31 15:04:20.525366 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.525328    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.527867 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.527841    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Jul 31 15:04:20.528191 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.528172    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Jul 31 15:04:20.528291 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.528219    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:04:20.528291 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.528223    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-2nxrd\""
Jul 31 15:04:20.528472 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.528244    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 31 15:04:20.529087 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.529071    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Jul 31 15:04:20.538704 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.538680    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"]
Jul 31 15:04:20.592595 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.592557    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-home\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.592781 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.592651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-model-cache\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.592781 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.592678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b16500e-44ec-493a-81be-5631a876a2fa-tls-certs\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.592781 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.592700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-kserve-provision-location\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.592781 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.592733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-tmp-dir\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.592999 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.592799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-dshm\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.592999 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.592917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b16500e-44ec-493a-81be-5631a876a2fa-cabundle-cert\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.592999 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.592955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnwjh\" (UniqueName: \"kubernetes.io/projected/4b16500e-44ec-493a-81be-5631a876a2fa-kube-api-access-mnwjh\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.693977 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.693942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b16500e-44ec-493a-81be-5631a876a2fa-tls-certs\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.693977 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.693981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-kserve-provision-location\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.694202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.694002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-tmp-dir\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.694202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.694024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-dshm\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.694202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.694057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b16500e-44ec-493a-81be-5631a876a2fa-cabundle-cert\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.694202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.694074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnwjh\" (UniqueName: \"kubernetes.io/projected/4b16500e-44ec-493a-81be-5631a876a2fa-kube-api-access-mnwjh\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.694202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.694099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-home\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.694202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.694130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-model-cache\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.694514 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.694449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-tmp-dir\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.694514 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.694481    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-kserve-provision-location\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.694631 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.694545    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-home\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.694725 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.694704    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-model-cache\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.694779 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.694759    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b16500e-44ec-493a-81be-5631a876a2fa-cabundle-cert\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.696426 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.696396    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-dshm\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.696696 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.696674    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b16500e-44ec-493a-81be-5631a876a2fa-tls-certs\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.703538 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.703514    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnwjh\" (UniqueName: \"kubernetes.io/projected/4b16500e-44ec-493a-81be-5631a876a2fa-kube-api-access-mnwjh\") pod \"auth-enabled-test-kserve-6fbcd9b9c-s8gvz\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.835988 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.835896    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:20.971156 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:20.971127    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"]
Jul 31 15:04:20.974541 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:04:20.974511    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b16500e_44ec_493a_81be_5631a876a2fa.slice/crio-b6ca8a779e5c249b3ed5a94c77c5277003c8576432a078feb9a9de4beb9525cd WatchSource:0}: Error finding container b6ca8a779e5c249b3ed5a94c77c5277003c8576432a078feb9a9de4beb9525cd: Status 404 returned error can't find the container with id b6ca8a779e5c249b3ed5a94c77c5277003c8576432a078feb9a9de4beb9525cd
Jul 31 15:04:21.190278 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:21.190199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" event={"ID":"4b16500e-44ec-493a-81be-5631a876a2fa","Type":"ContainerStarted","Data":"df9a6aee4c646d4cf4af90c60314aeefd598bdbab0ee64b1b12ae9c15fb79875"}
Jul 31 15:04:21.190278 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:21.190249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" event={"ID":"4b16500e-44ec-493a-81be-5631a876a2fa","Type":"ContainerStarted","Data":"b6ca8a779e5c249b3ed5a94c77c5277003c8576432a078feb9a9de4beb9525cd"}
Jul 31 15:04:23.198311 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:23.198281    2578 generic.go:356] "Generic (PLEG): container finished" podID="4b16500e-44ec-493a-81be-5631a876a2fa" containerID="df9a6aee4c646d4cf4af90c60314aeefd598bdbab0ee64b1b12ae9c15fb79875" exitCode=0
Jul 31 15:04:23.198653 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:23.198361    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" event={"ID":"4b16500e-44ec-493a-81be-5631a876a2fa","Type":"ContainerDied","Data":"df9a6aee4c646d4cf4af90c60314aeefd598bdbab0ee64b1b12ae9c15fb79875"}
Jul 31 15:04:24.204505 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:24.204470    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" event={"ID":"4b16500e-44ec-493a-81be-5631a876a2fa","Type":"ContainerStarted","Data":"390103a8ead7deee9bcac4be0a4337940bf5844d113678f84ab4ed572f81a691"}
Jul 31 15:04:24.229755 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:24.229689    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" podStartSLOduration=2.098991084 podStartE2EDuration="4.22967135s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.130680266s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:04:24.228326074 +0000 UTC m=+962.237537942" watchObservedRunningTime="2026-07-31 15:04:24.22967135 +0000 UTC m=+962.238883218"
Jul 31 15:04:24.836807 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:24.836772    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:26.799687 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:26.799648    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 31 15:04:30.836588 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:30.836545    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:04:30.838135 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:30.838093    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 31 15:04:40.836961 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:40.836913    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 31 15:04:41.821331 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:41.821284    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="53013629-4be2-4ce9-8ec3-6244e3467b9b" containerName="main" containerID="cri-o://8fb76bf1b6953101d590abc3a2524c234da8e5e625954940179f66e3cbbdcee2" gracePeriod=45
Jul 31 15:04:42.576556 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.576524    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:04:42.678383 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.678283    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53013629-4be2-4ce9-8ec3-6244e3467b9b-cabundle-cert\") pod \"53013629-4be2-4ce9-8ec3-6244e3467b9b\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") "
Jul 31 15:04:42.678383 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.678327    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-kserve-provision-location\") pod \"53013629-4be2-4ce9-8ec3-6244e3467b9b\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") "
Jul 31 15:04:42.678383 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.678368    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53013629-4be2-4ce9-8ec3-6244e3467b9b-tls-certs\") pod \"53013629-4be2-4ce9-8ec3-6244e3467b9b\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") "
Jul 31 15:04:42.678674 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.678409    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-dshm\") pod \"53013629-4be2-4ce9-8ec3-6244e3467b9b\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") "
Jul 31 15:04:42.678674 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.678437    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-tmp-dir\") pod \"53013629-4be2-4ce9-8ec3-6244e3467b9b\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") "
Jul 31 15:04:42.678674 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.678511    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-model-cache\") pod \"53013629-4be2-4ce9-8ec3-6244e3467b9b\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") "
Jul 31 15:04:42.678674 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.678547    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jwsb\" (UniqueName: \"kubernetes.io/projected/53013629-4be2-4ce9-8ec3-6244e3467b9b-kube-api-access-7jwsb\") pod \"53013629-4be2-4ce9-8ec3-6244e3467b9b\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") "
Jul 31 15:04:42.678674 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.678590    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-home\") pod \"53013629-4be2-4ce9-8ec3-6244e3467b9b\" (UID: \"53013629-4be2-4ce9-8ec3-6244e3467b9b\") "
Jul 31 15:04:42.678950 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.678683    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53013629-4be2-4ce9-8ec3-6244e3467b9b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "53013629-4be2-4ce9-8ec3-6244e3467b9b" (UID: "53013629-4be2-4ce9-8ec3-6244e3467b9b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:04:42.678950 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.678744    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "53013629-4be2-4ce9-8ec3-6244e3467b9b" (UID: "53013629-4be2-4ce9-8ec3-6244e3467b9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:42.678950 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.678850    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:42.678950 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.678868    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53013629-4be2-4ce9-8ec3-6244e3467b9b-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:42.679332 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.679304    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-home" (OuterVolumeSpecName: "home") pod "53013629-4be2-4ce9-8ec3-6244e3467b9b" (UID: "53013629-4be2-4ce9-8ec3-6244e3467b9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:42.680796 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.680764    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-dshm" (OuterVolumeSpecName: "dshm") pod "53013629-4be2-4ce9-8ec3-6244e3467b9b" (UID: "53013629-4be2-4ce9-8ec3-6244e3467b9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:42.680885 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.680850    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53013629-4be2-4ce9-8ec3-6244e3467b9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53013629-4be2-4ce9-8ec3-6244e3467b9b" (UID: "53013629-4be2-4ce9-8ec3-6244e3467b9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:04:42.680951 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.680933    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53013629-4be2-4ce9-8ec3-6244e3467b9b-kube-api-access-7jwsb" (OuterVolumeSpecName: "kube-api-access-7jwsb") pod "53013629-4be2-4ce9-8ec3-6244e3467b9b" (UID: "53013629-4be2-4ce9-8ec3-6244e3467b9b"). InnerVolumeSpecName "kube-api-access-7jwsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:04:42.691864 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.691841    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53013629-4be2-4ce9-8ec3-6244e3467b9b" (UID: "53013629-4be2-4ce9-8ec3-6244e3467b9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:42.705552 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.705528    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "53013629-4be2-4ce9-8ec3-6244e3467b9b" (UID: "53013629-4be2-4ce9-8ec3-6244e3467b9b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:42.779624 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.779586    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53013629-4be2-4ce9-8ec3-6244e3467b9b-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:42.779624 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.779618    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:42.779624 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.779630    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:42.779859 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.779642    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jwsb\" (UniqueName: \"kubernetes.io/projected/53013629-4be2-4ce9-8ec3-6244e3467b9b-kube-api-access-7jwsb\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:42.779859 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.779653    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:42.779859 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:42.779665    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53013629-4be2-4ce9-8ec3-6244e3467b9b-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:04:43.287941 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:43.287903    2578 generic.go:356] "Generic (PLEG): container finished" podID="53013629-4be2-4ce9-8ec3-6244e3467b9b" containerID="8fb76bf1b6953101d590abc3a2524c234da8e5e625954940179f66e3cbbdcee2" exitCode=0
Jul 31 15:04:43.288128 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:43.287955    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 31 15:04:43.288128 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:43.287989    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"53013629-4be2-4ce9-8ec3-6244e3467b9b","Type":"ContainerDied","Data":"8fb76bf1b6953101d590abc3a2524c234da8e5e625954940179f66e3cbbdcee2"}
Jul 31 15:04:43.288128 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:43.288040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"53013629-4be2-4ce9-8ec3-6244e3467b9b","Type":"ContainerDied","Data":"20476e3a9deccef468ad9a6ace7e4c9f54b29bab757b8320da1c86475282d741"}
Jul 31 15:04:43.288128 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:43.288061    2578 scope.go:117] "RemoveContainer" containerID="8fb76bf1b6953101d590abc3a2524c234da8e5e625954940179f66e3cbbdcee2"
Jul 31 15:04:43.297854 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:43.297836    2578 scope.go:117] "RemoveContainer" containerID="f9192c620c058f787085cb511dd2f05209027f7005f1023e92e1db934a79a709"
Jul 31 15:04:43.308139 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:43.308116    2578 scope.go:117] "RemoveContainer" containerID="8fb76bf1b6953101d590abc3a2524c234da8e5e625954940179f66e3cbbdcee2"
Jul 31 15:04:43.308468 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:04:43.308444    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb76bf1b6953101d590abc3a2524c234da8e5e625954940179f66e3cbbdcee2\": container with ID starting with 8fb76bf1b6953101d590abc3a2524c234da8e5e625954940179f66e3cbbdcee2 not found: ID does not exist" containerID="8fb76bf1b6953101d590abc3a2524c234da8e5e625954940179f66e3cbbdcee2"
Jul 31 15:04:43.308566 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:43.308474    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb76bf1b6953101d590abc3a2524c234da8e5e625954940179f66e3cbbdcee2"} err="failed to get container status \"8fb76bf1b6953101d590abc3a2524c234da8e5e625954940179f66e3cbbdcee2\": rpc error: code = NotFound desc = could not find container \"8fb76bf1b6953101d590abc3a2524c234da8e5e625954940179f66e3cbbdcee2\": container with ID starting with 8fb76bf1b6953101d590abc3a2524c234da8e5e625954940179f66e3cbbdcee2 not found: ID does not exist"
Jul 31 15:04:43.308566 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:43.308498    2578 scope.go:117] "RemoveContainer" containerID="f9192c620c058f787085cb511dd2f05209027f7005f1023e92e1db934a79a709"
Jul 31 15:04:43.308753 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:04:43.308737    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9192c620c058f787085cb511dd2f05209027f7005f1023e92e1db934a79a709\": container with ID starting with f9192c620c058f787085cb511dd2f05209027f7005f1023e92e1db934a79a709 not found: ID does not exist" containerID="f9192c620c058f787085cb511dd2f05209027f7005f1023e92e1db934a79a709"
Jul 31 15:04:43.308801 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:43.308756    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9192c620c058f787085cb511dd2f05209027f7005f1023e92e1db934a79a709"} err="failed to get container status \"f9192c620c058f787085cb511dd2f05209027f7005f1023e92e1db934a79a709\": rpc error: code = NotFound desc = could not find container \"f9192c620c058f787085cb511dd2f05209027f7005f1023e92e1db934a79a709\": container with ID starting with f9192c620c058f787085cb511dd2f05209027f7005f1023e92e1db934a79a709 not found: ID does not exist"
Jul 31 15:04:43.313446 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:43.313421    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 31 15:04:43.316705 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:43.316685    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 31 15:04:44.489407 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:44.489324    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53013629-4be2-4ce9-8ec3-6244e3467b9b" path="/var/lib/kubelet/pods/53013629-4be2-4ce9-8ec3-6244e3467b9b/volumes"
Jul 31 15:04:50.837048 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:50.836999    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 31 15:04:56.560483 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.560448    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"]
Jul 31 15:04:56.560861 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.560724    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53013629-4be2-4ce9-8ec3-6244e3467b9b" containerName="main"
Jul 31 15:04:56.560861 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.560736    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53013629-4be2-4ce9-8ec3-6244e3467b9b" containerName="main"
Jul 31 15:04:56.560861 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.560749    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53013629-4be2-4ce9-8ec3-6244e3467b9b" containerName="storage-initializer"
Jul 31 15:04:56.560861 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.560755    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53013629-4be2-4ce9-8ec3-6244e3467b9b" containerName="storage-initializer"
Jul 31 15:04:56.560861 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.560808    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="53013629-4be2-4ce9-8ec3-6244e3467b9b" containerName="main"
Jul 31 15:04:56.562599 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.562578    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"
Jul 31 15:04:56.568926 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.568899    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 31 15:04:56.569081 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.568899    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 31 15:04:56.569081 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.569042    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-ss9mt\""
Jul 31 15:04:56.569081 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.568902    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-g9k8k\""
Jul 31 15:04:56.569238 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.569173    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 31 15:04:56.574442 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.574418    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"]
Jul 31 15:04:56.577727 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.577697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf694\" (UniqueName: \"kubernetes.io/projected/6b253d77-17f9-458f-bc4f-b4654087fcc8-kube-api-access-cf694\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd\" (UID: \"6b253d77-17f9-458f-bc4f-b4654087fcc8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"
Jul 31 15:04:56.577849 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.577760    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b253d77-17f9-458f-bc4f-b4654087fcc8-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd\" (UID: \"6b253d77-17f9-458f-bc4f-b4654087fcc8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"
Jul 31 15:04:56.678314 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.678278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf694\" (UniqueName: \"kubernetes.io/projected/6b253d77-17f9-458f-bc4f-b4654087fcc8-kube-api-access-cf694\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd\" (UID: \"6b253d77-17f9-458f-bc4f-b4654087fcc8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"
Jul 31 15:04:56.678314 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.678327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b253d77-17f9-458f-bc4f-b4654087fcc8-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd\" (UID: \"6b253d77-17f9-458f-bc4f-b4654087fcc8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"
Jul 31 15:04:56.680793 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.680766    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b253d77-17f9-458f-bc4f-b4654087fcc8-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd\" (UID: \"6b253d77-17f9-458f-bc4f-b4654087fcc8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"
Jul 31 15:04:56.687604 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.687577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf694\" (UniqueName: \"kubernetes.io/projected/6b253d77-17f9-458f-bc4f-b4654087fcc8-kube-api-access-cf694\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd\" (UID: \"6b253d77-17f9-458f-bc4f-b4654087fcc8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"
Jul 31 15:04:56.874334 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:56.874251    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"
Jul 31 15:04:57.011727 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:57.011664    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"]
Jul 31 15:04:57.014143 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:04:57.014099    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b253d77_17f9_458f_bc4f_b4654087fcc8.slice/crio-ea409bb38f3cf089efd4355bfb53d6f3a516dd7bd36062c3ec905da136315090 WatchSource:0}: Error finding container ea409bb38f3cf089efd4355bfb53d6f3a516dd7bd36062c3ec905da136315090: Status 404 returned error can't find the container with id ea409bb38f3cf089efd4355bfb53d6f3a516dd7bd36062c3ec905da136315090
Jul 31 15:04:57.352112 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:57.352063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd" event={"ID":"6b253d77-17f9-458f-bc4f-b4654087fcc8","Type":"ContainerStarted","Data":"c3a3ef3e395ce6ea07a61e996e9ded683a13dab01505ea06412cc0d4302ef68a"}
Jul 31 15:04:57.352306 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:57.352142    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd" event={"ID":"6b253d77-17f9-458f-bc4f-b4654087fcc8","Type":"ContainerStarted","Data":"ea409bb38f3cf089efd4355bfb53d6f3a516dd7bd36062c3ec905da136315090"}
Jul 31 15:04:57.352407 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:57.352319    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"
Jul 31 15:04:57.372561 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:04:57.372495    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd" podStartSLOduration=1.372471554 podStartE2EDuration="1.372471554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:04:57.370496858 +0000 UTC m=+995.379708730" watchObservedRunningTime="2026-07-31 15:04:57.372471554 +0000 UTC m=+995.381683424"
Jul 31 15:05:00.837343 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:05:00.837292    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 31 15:05:10.837450 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:05:10.836867    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 31 15:05:20.836590 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:05:20.836548    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 31 15:05:28.359022 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:05:28.358988    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"
Jul 31 15:05:30.837043 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:05:30.836988    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 31 15:05:40.836718 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:05:40.836663    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 31 15:05:44.723626 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:05:44.723588    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"]
Jul 31 15:05:48.359712 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:05:48.359673    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd" podUID="6b253d77-17f9-458f-bc4f-b4654087fcc8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 15:05:50.836525 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:05:50.836482    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 31 15:05:58.359414 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:05:58.359368    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd" podUID="6b253d77-17f9-458f-bc4f-b4654087fcc8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 15:05:59.743174 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:05:59.743111    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd" podUID="6b253d77-17f9-458f-bc4f-b4654087fcc8" containerName="main" containerID="cri-o://c3a3ef3e395ce6ea07a61e996e9ded683a13dab01505ea06412cc0d4302ef68a" gracePeriod=45
Jul 31 15:05:59.975905 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:05:59.975873    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"
Jul 31 15:06:00.102745 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:00.102644    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b253d77-17f9-458f-bc4f-b4654087fcc8-tls-certs\") pod \"6b253d77-17f9-458f-bc4f-b4654087fcc8\" (UID: \"6b253d77-17f9-458f-bc4f-b4654087fcc8\") "
Jul 31 15:06:00.102913 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:00.102766    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf694\" (UniqueName: \"kubernetes.io/projected/6b253d77-17f9-458f-bc4f-b4654087fcc8-kube-api-access-cf694\") pod \"6b253d77-17f9-458f-bc4f-b4654087fcc8\" (UID: \"6b253d77-17f9-458f-bc4f-b4654087fcc8\") "
Jul 31 15:06:00.104997 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:00.104963    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b253d77-17f9-458f-bc4f-b4654087fcc8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b253d77-17f9-458f-bc4f-b4654087fcc8" (UID: "6b253d77-17f9-458f-bc4f-b4654087fcc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:06:00.105134 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:00.104967    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b253d77-17f9-458f-bc4f-b4654087fcc8-kube-api-access-cf694" (OuterVolumeSpecName: "kube-api-access-cf694") pod "6b253d77-17f9-458f-bc4f-b4654087fcc8" (UID: "6b253d77-17f9-458f-bc4f-b4654087fcc8"). InnerVolumeSpecName "kube-api-access-cf694". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:06:00.203930 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:00.203889    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf694\" (UniqueName: \"kubernetes.io/projected/6b253d77-17f9-458f-bc4f-b4654087fcc8-kube-api-access-cf694\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:06:00.203930 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:00.203920    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b253d77-17f9-458f-bc4f-b4654087fcc8-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:06:00.616503 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:00.616471    2578 generic.go:356] "Generic (PLEG): container finished" podID="6b253d77-17f9-458f-bc4f-b4654087fcc8" containerID="c3a3ef3e395ce6ea07a61e996e9ded683a13dab01505ea06412cc0d4302ef68a" exitCode=0
Jul 31 15:06:00.616688 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:00.616512    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"
Jul 31 15:06:00.616688 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:00.616525    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd" event={"ID":"6b253d77-17f9-458f-bc4f-b4654087fcc8","Type":"ContainerDied","Data":"c3a3ef3e395ce6ea07a61e996e9ded683a13dab01505ea06412cc0d4302ef68a"}
Jul 31 15:06:00.616688 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:00.616567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd" event={"ID":"6b253d77-17f9-458f-bc4f-b4654087fcc8","Type":"ContainerDied","Data":"ea409bb38f3cf089efd4355bfb53d6f3a516dd7bd36062c3ec905da136315090"}
Jul 31 15:06:00.616688 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:00.616588    2578 scope.go:117] "RemoveContainer" containerID="c3a3ef3e395ce6ea07a61e996e9ded683a13dab01505ea06412cc0d4302ef68a"
Jul 31 15:06:00.624795 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:00.624774    2578 scope.go:117] "RemoveContainer" containerID="c3a3ef3e395ce6ea07a61e996e9ded683a13dab01505ea06412cc0d4302ef68a"
Jul 31 15:06:00.625079 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:06:00.625040    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3a3ef3e395ce6ea07a61e996e9ded683a13dab01505ea06412cc0d4302ef68a\": container with ID starting with c3a3ef3e395ce6ea07a61e996e9ded683a13dab01505ea06412cc0d4302ef68a not found: ID does not exist" containerID="c3a3ef3e395ce6ea07a61e996e9ded683a13dab01505ea06412cc0d4302ef68a"
Jul 31 15:06:00.625162 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:00.625085    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a3ef3e395ce6ea07a61e996e9ded683a13dab01505ea06412cc0d4302ef68a"} err="failed to get container status \"c3a3ef3e395ce6ea07a61e996e9ded683a13dab01505ea06412cc0d4302ef68a\": rpc error: code = NotFound desc = could not find container \"c3a3ef3e395ce6ea07a61e996e9ded683a13dab01505ea06412cc0d4302ef68a\": container with ID starting with c3a3ef3e395ce6ea07a61e996e9ded683a13dab01505ea06412cc0d4302ef68a not found: ID does not exist"
Jul 31 15:06:00.634322 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:00.634292    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"]
Jul 31 15:06:00.638667 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:00.638643    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5p8jcd"]
Jul 31 15:06:00.837028 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:00.836918    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 31 15:06:02.489428 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:02.489390    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b253d77-17f9-458f-bc4f-b4654087fcc8" path="/var/lib/kubelet/pods/6b253d77-17f9-458f-bc4f-b4654087fcc8/volumes"
Jul 31 15:06:10.837302 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:10.837253    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 31 15:06:13.733053 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:13.733016    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"]
Jul 31 15:06:13.733432 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:13.733321    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b253d77-17f9-458f-bc4f-b4654087fcc8" containerName="main"
Jul 31 15:06:13.733432 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:13.733337    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b253d77-17f9-458f-bc4f-b4654087fcc8" containerName="main"
Jul 31 15:06:13.733432 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:13.733421    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b253d77-17f9-458f-bc4f-b4654087fcc8" containerName="main"
Jul 31 15:06:13.738206 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:13.738186    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"
Jul 31 15:06:13.743254 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:13.743220    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-gh4m7\""
Jul 31 15:06:13.743401 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:13.743230    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 31 15:06:13.743401 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:13.743289    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-dnnq5\""
Jul 31 15:06:13.743513 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:13.743456    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Jul 31 15:06:13.744257 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:13.744242    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Jul 31 15:06:13.748311 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:13.748282    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"]
Jul 31 15:06:13.812082 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:13.812048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm4b5\" (UniqueName: \"kubernetes.io/projected/ede3458a-2748-4b47-9b99-cb43b2cfd81e-kube-api-access-zm4b5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr\" (UID: \"ede3458a-2748-4b47-9b99-cb43b2cfd81e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"
Jul 31 15:06:13.812263 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:13.812153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede3458a-2748-4b47-9b99-cb43b2cfd81e-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr\" (UID: \"ede3458a-2748-4b47-9b99-cb43b2cfd81e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"
Jul 31 15:06:13.912659 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:13.912612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede3458a-2748-4b47-9b99-cb43b2cfd81e-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr\" (UID: \"ede3458a-2748-4b47-9b99-cb43b2cfd81e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"
Jul 31 15:06:13.912827 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:13.912696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm4b5\" (UniqueName: \"kubernetes.io/projected/ede3458a-2748-4b47-9b99-cb43b2cfd81e-kube-api-access-zm4b5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr\" (UID: \"ede3458a-2748-4b47-9b99-cb43b2cfd81e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"
Jul 31 15:06:13.915077 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:13.915048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede3458a-2748-4b47-9b99-cb43b2cfd81e-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr\" (UID: \"ede3458a-2748-4b47-9b99-cb43b2cfd81e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"
Jul 31 15:06:13.921719 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:13.921697    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm4b5\" (UniqueName: \"kubernetes.io/projected/ede3458a-2748-4b47-9b99-cb43b2cfd81e-kube-api-access-zm4b5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr\" (UID: \"ede3458a-2748-4b47-9b99-cb43b2cfd81e\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"
Jul 31 15:06:14.049236 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:14.049149    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"
Jul 31 15:06:14.175595 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:14.175567    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"]
Jul 31 15:06:14.177817 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:06:14.177778    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede3458a_2748_4b47_9b99_cb43b2cfd81e.slice/crio-3b2bd564519c2521f505a17a42ef504730ba302f4cd807373c280f557d1d5d0e WatchSource:0}: Error finding container 3b2bd564519c2521f505a17a42ef504730ba302f4cd807373c280f557d1d5d0e: Status 404 returned error can't find the container with id 3b2bd564519c2521f505a17a42ef504730ba302f4cd807373c280f557d1d5d0e
Jul 31 15:06:14.679580 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:14.679543    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr" event={"ID":"ede3458a-2748-4b47-9b99-cb43b2cfd81e","Type":"ContainerStarted","Data":"9609674bb6fdd6f8aae595f265c7468bb1940e3d31e78ac87e9b49418df2a880"}
Jul 31 15:06:14.679789 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:14.679599    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr" event={"ID":"ede3458a-2748-4b47-9b99-cb43b2cfd81e","Type":"ContainerStarted","Data":"3b2bd564519c2521f505a17a42ef504730ba302f4cd807373c280f557d1d5d0e"}
Jul 31 15:06:14.679789 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:14.679633    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"
Jul 31 15:06:14.699987 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:14.699932    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr" podStartSLOduration=1.699916918 podStartE2EDuration="1.699916918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:06:14.698243798 +0000 UTC m=+1072.707455680" watchObservedRunningTime="2026-07-31 15:06:14.699916918 +0000 UTC m=+1072.709128786"
Jul 31 15:06:20.846490 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:20.846450    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:06:20.854346 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:20.854319    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:06:25.394433 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:25.394399    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"]
Jul 31 15:06:40.416185 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:40.416119    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="main" containerID="cri-o://390103a8ead7deee9bcac4be0a4337940bf5844d113678f84ab4ed572f81a691" gracePeriod=45
Jul 31 15:06:40.848101 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:40.848048    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 31 15:06:41.161219 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.161189    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:06:41.252016 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.251978    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-dshm\") pod \"4b16500e-44ec-493a-81be-5631a876a2fa\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") "
Jul 31 15:06:41.252016 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.252020    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-model-cache\") pod \"4b16500e-44ec-493a-81be-5631a876a2fa\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") "
Jul 31 15:06:41.252240 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.252044    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b16500e-44ec-493a-81be-5631a876a2fa-tls-certs\") pod \"4b16500e-44ec-493a-81be-5631a876a2fa\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") "
Jul 31 15:06:41.252240 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.252076    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-kserve-provision-location\") pod \"4b16500e-44ec-493a-81be-5631a876a2fa\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") "
Jul 31 15:06:41.252240 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.252109    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-tmp-dir\") pod \"4b16500e-44ec-493a-81be-5631a876a2fa\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") "
Jul 31 15:06:41.252240 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.252160    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-home\") pod \"4b16500e-44ec-493a-81be-5631a876a2fa\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") "
Jul 31 15:06:41.252240 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.252198    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b16500e-44ec-493a-81be-5631a876a2fa-cabundle-cert\") pod \"4b16500e-44ec-493a-81be-5631a876a2fa\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") "
Jul 31 15:06:41.252526 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.252246    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnwjh\" (UniqueName: \"kubernetes.io/projected/4b16500e-44ec-493a-81be-5631a876a2fa-kube-api-access-mnwjh\") pod \"4b16500e-44ec-493a-81be-5631a876a2fa\" (UID: \"4b16500e-44ec-493a-81be-5631a876a2fa\") "
Jul 31 15:06:41.252526 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.252325    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b16500e-44ec-493a-81be-5631a876a2fa" (UID: "4b16500e-44ec-493a-81be-5631a876a2fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:06:41.252526 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.252506    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:06:41.252672 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.252608    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b16500e-44ec-493a-81be-5631a876a2fa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4b16500e-44ec-493a-81be-5631a876a2fa" (UID: "4b16500e-44ec-493a-81be-5631a876a2fa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:06:41.253008 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.252982    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-home" (OuterVolumeSpecName: "home") pod "4b16500e-44ec-493a-81be-5631a876a2fa" (UID: "4b16500e-44ec-493a-81be-5631a876a2fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:06:41.254442 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.254420    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-dshm" (OuterVolumeSpecName: "dshm") pod "4b16500e-44ec-493a-81be-5631a876a2fa" (UID: "4b16500e-44ec-493a-81be-5631a876a2fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:06:41.254760 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.254736    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b16500e-44ec-493a-81be-5631a876a2fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b16500e-44ec-493a-81be-5631a876a2fa" (UID: "4b16500e-44ec-493a-81be-5631a876a2fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:06:41.254832 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.254754    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b16500e-44ec-493a-81be-5631a876a2fa-kube-api-access-mnwjh" (OuterVolumeSpecName: "kube-api-access-mnwjh") pod "4b16500e-44ec-493a-81be-5631a876a2fa" (UID: "4b16500e-44ec-493a-81be-5631a876a2fa"). InnerVolumeSpecName "kube-api-access-mnwjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:06:41.271302 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.271269    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b16500e-44ec-493a-81be-5631a876a2fa" (UID: "4b16500e-44ec-493a-81be-5631a876a2fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:06:41.279648 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.279623    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b16500e-44ec-493a-81be-5631a876a2fa" (UID: "4b16500e-44ec-493a-81be-5631a876a2fa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:06:41.353231 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.353192    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:06:41.353231 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.353225    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b16500e-44ec-493a-81be-5631a876a2fa-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:06:41.353231 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.353237    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnwjh\" (UniqueName: \"kubernetes.io/projected/4b16500e-44ec-493a-81be-5631a876a2fa-kube-api-access-mnwjh\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:06:41.353500 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.353248    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:06:41.353500 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.353257    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b16500e-44ec-493a-81be-5631a876a2fa-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:06:41.353500 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.353266    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:06:41.353500 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.353274    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b16500e-44ec-493a-81be-5631a876a2fa-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:06:41.788128 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.788093    2578 generic.go:356] "Generic (PLEG): container finished" podID="4b16500e-44ec-493a-81be-5631a876a2fa" containerID="390103a8ead7deee9bcac4be0a4337940bf5844d113678f84ab4ed572f81a691" exitCode=0
Jul 31 15:06:41.788711 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.788150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" event={"ID":"4b16500e-44ec-493a-81be-5631a876a2fa","Type":"ContainerDied","Data":"390103a8ead7deee9bcac4be0a4337940bf5844d113678f84ab4ed572f81a691"}
Jul 31 15:06:41.788711 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.788160    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"
Jul 31 15:06:41.788711 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.788213    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz" event={"ID":"4b16500e-44ec-493a-81be-5631a876a2fa","Type":"ContainerDied","Data":"b6ca8a779e5c249b3ed5a94c77c5277003c8576432a078feb9a9de4beb9525cd"}
Jul 31 15:06:41.788711 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.788235    2578 scope.go:117] "RemoveContainer" containerID="390103a8ead7deee9bcac4be0a4337940bf5844d113678f84ab4ed572f81a691"
Jul 31 15:06:41.796955 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.796937    2578 scope.go:117] "RemoveContainer" containerID="df9a6aee4c646d4cf4af90c60314aeefd598bdbab0ee64b1b12ae9c15fb79875"
Jul 31 15:06:41.804074 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.804053    2578 scope.go:117] "RemoveContainer" containerID="390103a8ead7deee9bcac4be0a4337940bf5844d113678f84ab4ed572f81a691"
Jul 31 15:06:41.804366 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:06:41.804330    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390103a8ead7deee9bcac4be0a4337940bf5844d113678f84ab4ed572f81a691\": container with ID starting with 390103a8ead7deee9bcac4be0a4337940bf5844d113678f84ab4ed572f81a691 not found: ID does not exist" containerID="390103a8ead7deee9bcac4be0a4337940bf5844d113678f84ab4ed572f81a691"
Jul 31 15:06:41.804418 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.804373    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390103a8ead7deee9bcac4be0a4337940bf5844d113678f84ab4ed572f81a691"} err="failed to get container status \"390103a8ead7deee9bcac4be0a4337940bf5844d113678f84ab4ed572f81a691\": rpc error: code = NotFound desc = could not find container \"390103a8ead7deee9bcac4be0a4337940bf5844d113678f84ab4ed572f81a691\": container with ID starting with 390103a8ead7deee9bcac4be0a4337940bf5844d113678f84ab4ed572f81a691 not found: ID does not exist"
Jul 31 15:06:41.804418 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.804397    2578 scope.go:117] "RemoveContainer" containerID="df9a6aee4c646d4cf4af90c60314aeefd598bdbab0ee64b1b12ae9c15fb79875"
Jul 31 15:06:41.804622 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:06:41.804605    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df9a6aee4c646d4cf4af90c60314aeefd598bdbab0ee64b1b12ae9c15fb79875\": container with ID starting with df9a6aee4c646d4cf4af90c60314aeefd598bdbab0ee64b1b12ae9c15fb79875 not found: ID does not exist" containerID="df9a6aee4c646d4cf4af90c60314aeefd598bdbab0ee64b1b12ae9c15fb79875"
Jul 31 15:06:41.804661 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.804627    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df9a6aee4c646d4cf4af90c60314aeefd598bdbab0ee64b1b12ae9c15fb79875"} err="failed to get container status \"df9a6aee4c646d4cf4af90c60314aeefd598bdbab0ee64b1b12ae9c15fb79875\": rpc error: code = NotFound desc = could not find container \"df9a6aee4c646d4cf4af90c60314aeefd598bdbab0ee64b1b12ae9c15fb79875\": container with ID starting with df9a6aee4c646d4cf4af90c60314aeefd598bdbab0ee64b1b12ae9c15fb79875 not found: ID does not exist"
Jul 31 15:06:41.816879 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.816855    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"]
Jul 31 15:06:41.822923 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:41.822903    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6fbcd9b9c-s8gvz"]
Jul 31 15:06:42.488968 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:42.488937    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" path="/var/lib/kubelet/pods/4b16500e-44ec-493a-81be-5631a876a2fa/volumes"
Jul 31 15:06:45.685015 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:45.684987    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"
Jul 31 15:06:55.401114 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.401074    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"]
Jul 31 15:06:55.401599 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.401493    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="storage-initializer"
Jul 31 15:06:55.401599 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.401511    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="storage-initializer"
Jul 31 15:06:55.401599 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.401522    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="main"
Jul 31 15:06:55.401599 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.401529    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="main"
Jul 31 15:06:55.401807 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.401603    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b16500e-44ec-493a-81be-5631a876a2fa" containerName="main"
Jul 31 15:06:55.404535 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.404513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.407618 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.407597    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Jul 31 15:06:55.407954 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.407934    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Jul 31 15:06:55.407954 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.407964    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Jul 31 15:06:55.408200 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.408178    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-xrdkg\""
Jul 31 15:06:55.408297 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.408199    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:06:55.408297 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.408212    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 31 15:06:55.422910 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.422889    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"]
Jul 31 15:06:55.567234 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.567197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.567234 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.567234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27cb2126-dc8c-4659-9215-4f5a0ff9947e-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.567483 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.567260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf4bf\" (UniqueName: \"kubernetes.io/projected/27cb2126-dc8c-4659-9215-4f5a0ff9947e-kube-api-access-tf4bf\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.567483 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.567322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-home\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.567483 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.567361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-dshm\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.567483 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.567411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-tmp-dir\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.567483 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.567468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-model-cache\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.567678 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.567492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27cb2126-dc8c-4659-9215-4f5a0ff9947e-tls-certs\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.668624 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.668537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27cb2126-dc8c-4659-9215-4f5a0ff9947e-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.668624 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.668583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf4bf\" (UniqueName: \"kubernetes.io/projected/27cb2126-dc8c-4659-9215-4f5a0ff9947e-kube-api-access-tf4bf\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.668624 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.668607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-home\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.668905 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.668628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-dshm\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.668905 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.668652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-tmp-dir\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.668905 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.668689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-model-cache\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.668905 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.668719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27cb2126-dc8c-4659-9215-4f5a0ff9947e-tls-certs\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.668905 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.668768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.669185 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.668989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-home\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.669185 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.669017    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-model-cache\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.669185 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.669143    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.669185 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.669177    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-tmp-dir\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.669460 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.669433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27cb2126-dc8c-4659-9215-4f5a0ff9947e-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.671380 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.671335    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27cb2126-dc8c-4659-9215-4f5a0ff9947e-tls-certs\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.671537 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.671519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-dshm\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.684031 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.684006    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf4bf\" (UniqueName: \"kubernetes.io/projected/27cb2126-dc8c-4659-9215-4f5a0ff9947e-kube-api-access-tf4bf\") pod \"auth-invalid-token-test-kserve-5ffd754788-qnrjr\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.714878 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.714855    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:06:55.853898 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:55.853872    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"]
Jul 31 15:06:55.856177 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:06:55.856140    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cb2126_dc8c_4659_9215_4f5a0ff9947e.slice/crio-0c083c52e8b12d35f8273526d3a996671c23d21af3099b4dfaec77b558e75d05 WatchSource:0}: Error finding container 0c083c52e8b12d35f8273526d3a996671c23d21af3099b4dfaec77b558e75d05: Status 404 returned error can't find the container with id 0c083c52e8b12d35f8273526d3a996671c23d21af3099b4dfaec77b558e75d05
Jul 31 15:06:56.850289 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:56.850241    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" event={"ID":"27cb2126-dc8c-4659-9215-4f5a0ff9947e","Type":"ContainerStarted","Data":"457f25fba6f4d2057a66929fb35153e9d62f5dff93147eb31da0a65a8dfa222b"}
Jul 31 15:06:56.850797 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:56.850320    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" event={"ID":"27cb2126-dc8c-4659-9215-4f5a0ff9947e","Type":"ContainerStarted","Data":"0c083c52e8b12d35f8273526d3a996671c23d21af3099b4dfaec77b558e75d05"}
Jul 31 15:06:57.926290 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:57.926256    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"]
Jul 31 15:06:58.861777 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:58.861740    2578 generic.go:356] "Generic (PLEG): container finished" podID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerID="457f25fba6f4d2057a66929fb35153e9d62f5dff93147eb31da0a65a8dfa222b" exitCode=0
Jul 31 15:06:58.861981 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:58.861798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" event={"ID":"27cb2126-dc8c-4659-9215-4f5a0ff9947e","Type":"ContainerDied","Data":"457f25fba6f4d2057a66929fb35153e9d62f5dff93147eb31da0a65a8dfa222b"}
Jul 31 15:06:59.867252 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:59.867215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" event={"ID":"27cb2126-dc8c-4659-9215-4f5a0ff9947e","Type":"ContainerStarted","Data":"adbac8f524192a6243302d33401cea2ca56dec5201a271f5ed88390e934c6952"}
Jul 31 15:06:59.893812 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:06:59.893763    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" podStartSLOduration=2.69482565 podStartE2EDuration="4.893744616s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.198918966s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:06:59.893181139 +0000 UTC m=+1117.902393010" watchObservedRunningTime="2026-07-31 15:06:59.893744616 +0000 UTC m=+1117.902956487"
Jul 31 15:07:00.715513 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:00.715471    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:07:05.691126 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:05.691076    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr" podUID="ede3458a-2748-4b47-9b99-cb43b2cfd81e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 15:07:05.715103 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:05.715072    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:07:05.716584 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:05.716560    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 31 15:07:12.948333 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:12.948275    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr" podUID="ede3458a-2748-4b47-9b99-cb43b2cfd81e" containerName="main" containerID="cri-o://9609674bb6fdd6f8aae595f265c7468bb1940e3d31e78ac87e9b49418df2a880" gracePeriod=45
Jul 31 15:07:13.184813 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:13.184786    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"
Jul 31 15:07:13.210299 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:13.210221    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm4b5\" (UniqueName: \"kubernetes.io/projected/ede3458a-2748-4b47-9b99-cb43b2cfd81e-kube-api-access-zm4b5\") pod \"ede3458a-2748-4b47-9b99-cb43b2cfd81e\" (UID: \"ede3458a-2748-4b47-9b99-cb43b2cfd81e\") "
Jul 31 15:07:13.210478 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:13.210314    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede3458a-2748-4b47-9b99-cb43b2cfd81e-tls-certs\") pod \"ede3458a-2748-4b47-9b99-cb43b2cfd81e\" (UID: \"ede3458a-2748-4b47-9b99-cb43b2cfd81e\") "
Jul 31 15:07:13.212470 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:13.212442    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede3458a-2748-4b47-9b99-cb43b2cfd81e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ede3458a-2748-4b47-9b99-cb43b2cfd81e" (UID: "ede3458a-2748-4b47-9b99-cb43b2cfd81e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:07:13.212609 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:13.212516    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede3458a-2748-4b47-9b99-cb43b2cfd81e-kube-api-access-zm4b5" (OuterVolumeSpecName: "kube-api-access-zm4b5") pod "ede3458a-2748-4b47-9b99-cb43b2cfd81e" (UID: "ede3458a-2748-4b47-9b99-cb43b2cfd81e"). InnerVolumeSpecName "kube-api-access-zm4b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:07:13.311266 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:13.311219    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede3458a-2748-4b47-9b99-cb43b2cfd81e-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:07:13.311266 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:13.311250    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm4b5\" (UniqueName: \"kubernetes.io/projected/ede3458a-2748-4b47-9b99-cb43b2cfd81e-kube-api-access-zm4b5\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:07:13.930313 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:13.930273    2578 generic.go:356] "Generic (PLEG): container finished" podID="ede3458a-2748-4b47-9b99-cb43b2cfd81e" containerID="9609674bb6fdd6f8aae595f265c7468bb1940e3d31e78ac87e9b49418df2a880" exitCode=0
Jul 31 15:07:13.930313 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:13.930314    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"
Jul 31 15:07:13.930539 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:13.930365    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr" event={"ID":"ede3458a-2748-4b47-9b99-cb43b2cfd81e","Type":"ContainerDied","Data":"9609674bb6fdd6f8aae595f265c7468bb1940e3d31e78ac87e9b49418df2a880"}
Jul 31 15:07:13.930539 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:13.930420    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr" event={"ID":"ede3458a-2748-4b47-9b99-cb43b2cfd81e","Type":"ContainerDied","Data":"3b2bd564519c2521f505a17a42ef504730ba302f4cd807373c280f557d1d5d0e"}
Jul 31 15:07:13.930539 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:13.930447    2578 scope.go:117] "RemoveContainer" containerID="9609674bb6fdd6f8aae595f265c7468bb1940e3d31e78ac87e9b49418df2a880"
Jul 31 15:07:13.938727 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:13.938706    2578 scope.go:117] "RemoveContainer" containerID="9609674bb6fdd6f8aae595f265c7468bb1940e3d31e78ac87e9b49418df2a880"
Jul 31 15:07:13.939004 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:07:13.938980    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9609674bb6fdd6f8aae595f265c7468bb1940e3d31e78ac87e9b49418df2a880\": container with ID starting with 9609674bb6fdd6f8aae595f265c7468bb1940e3d31e78ac87e9b49418df2a880 not found: ID does not exist" containerID="9609674bb6fdd6f8aae595f265c7468bb1940e3d31e78ac87e9b49418df2a880"
Jul 31 15:07:13.939056 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:13.939012    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9609674bb6fdd6f8aae595f265c7468bb1940e3d31e78ac87e9b49418df2a880"} err="failed to get container status \"9609674bb6fdd6f8aae595f265c7468bb1940e3d31e78ac87e9b49418df2a880\": rpc error: code = NotFound desc = could not find container \"9609674bb6fdd6f8aae595f265c7468bb1940e3d31e78ac87e9b49418df2a880\": container with ID starting with 9609674bb6fdd6f8aae595f265c7468bb1940e3d31e78ac87e9b49418df2a880 not found: ID does not exist"
Jul 31 15:07:13.952991 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:13.952958    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"]
Jul 31 15:07:13.958924 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:13.958899    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehthmr"]
Jul 31 15:07:14.491048 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:14.491012    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ede3458a-2748-4b47-9b99-cb43b2cfd81e" path="/var/lib/kubelet/pods/ede3458a-2748-4b47-9b99-cb43b2cfd81e/volumes"
Jul 31 15:07:15.716167 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:15.716119    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 31 15:07:21.415221 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.415176    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"]
Jul 31 15:07:21.415710 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.415628    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ede3458a-2748-4b47-9b99-cb43b2cfd81e" containerName="main"
Jul 31 15:07:21.415710 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.415648    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ede3458a-2748-4b47-9b99-cb43b2cfd81e" containerName="main"
Jul 31 15:07:21.415792 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.415733    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ede3458a-2748-4b47-9b99-cb43b2cfd81e" containerName="main"
Jul 31 15:07:21.417696 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.417676    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"
Jul 31 15:07:21.420691 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.420668    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Jul 31 15:07:21.421810 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.421791    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Jul 31 15:07:21.421922 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.421870    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 31 15:07:21.422067 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.422042    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-rlmnl\""
Jul 31 15:07:21.422186 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.422122    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-bhvm9\""
Jul 31 15:07:21.427898 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.427870    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"]
Jul 31 15:07:21.480714 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.480674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2a59049-7024-4284-9c8c-8cd039bab9ec-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx\" (UID: \"d2a59049-7024-4284-9c8c-8cd039bab9ec\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"
Jul 31 15:07:21.480893 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.480734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2wnv\" (UniqueName: \"kubernetes.io/projected/d2a59049-7024-4284-9c8c-8cd039bab9ec-kube-api-access-r2wnv\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx\" (UID: \"d2a59049-7024-4284-9c8c-8cd039bab9ec\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"
Jul 31 15:07:21.581701 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.581666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2wnv\" (UniqueName: \"kubernetes.io/projected/d2a59049-7024-4284-9c8c-8cd039bab9ec-kube-api-access-r2wnv\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx\" (UID: \"d2a59049-7024-4284-9c8c-8cd039bab9ec\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"
Jul 31 15:07:21.581850 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.581758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2a59049-7024-4284-9c8c-8cd039bab9ec-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx\" (UID: \"d2a59049-7024-4284-9c8c-8cd039bab9ec\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"
Jul 31 15:07:21.584418 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.584388    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2a59049-7024-4284-9c8c-8cd039bab9ec-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx\" (UID: \"d2a59049-7024-4284-9c8c-8cd039bab9ec\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"
Jul 31 15:07:21.590090 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.590060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2wnv\" (UniqueName: \"kubernetes.io/projected/d2a59049-7024-4284-9c8c-8cd039bab9ec-kube-api-access-r2wnv\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx\" (UID: \"d2a59049-7024-4284-9c8c-8cd039bab9ec\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"
Jul 31 15:07:21.730244 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.730208    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"
Jul 31 15:07:21.865550 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.865524    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"]
Jul 31 15:07:21.867894 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:07:21.867857    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a59049_7024_4284_9c8c_8cd039bab9ec.slice/crio-aa542a72d298e073283c1cf893cf38e5752655ded4e2cc14e86ca9ae021be8a7 WatchSource:0}: Error finding container aa542a72d298e073283c1cf893cf38e5752655ded4e2cc14e86ca9ae021be8a7: Status 404 returned error can't find the container with id aa542a72d298e073283c1cf893cf38e5752655ded4e2cc14e86ca9ae021be8a7
Jul 31 15:07:21.965343 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.965305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx" event={"ID":"d2a59049-7024-4284-9c8c-8cd039bab9ec","Type":"ContainerStarted","Data":"82e9aecab4e9c248f80531299d5d3a95d3e0abb8407a414a96b65ac3c5cf5df7"}
Jul 31 15:07:21.965525 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.965403    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"
Jul 31 15:07:21.965525 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:21.965433    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx" event={"ID":"d2a59049-7024-4284-9c8c-8cd039bab9ec","Type":"ContainerStarted","Data":"aa542a72d298e073283c1cf893cf38e5752655ded4e2cc14e86ca9ae021be8a7"}
Jul 31 15:07:22.006073 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:22.005969    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx" podStartSLOduration=1.005950113 podStartE2EDuration="1.005950113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:07:22.005332039 +0000 UTC m=+1140.014543908" watchObservedRunningTime="2026-07-31 15:07:22.005950113 +0000 UTC m=+1140.015161982"
Jul 31 15:07:25.716110 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:25.716045    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 31 15:07:35.715716 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:35.715666    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 31 15:07:45.715577 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:45.715476    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 31 15:07:52.971694 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:52.971656    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"
Jul 31 15:07:54.425732 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:54.425698    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"]
Jul 31 15:07:55.716368 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:07:55.716302    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 31 15:08:02.971102 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:02.971056    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx" podUID="d2a59049-7024-4284-9c8c-8cd039bab9ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 15:08:05.716364 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:05.716299    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 31 15:08:09.444087 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:09.444028    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx" podUID="d2a59049-7024-4284-9c8c-8cd039bab9ec" containerName="main" containerID="cri-o://82e9aecab4e9c248f80531299d5d3a95d3e0abb8407a414a96b65ac3c5cf5df7" gracePeriod=45
Jul 31 15:08:09.678948 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:09.678925    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"
Jul 31 15:08:09.781642 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:09.781608    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2wnv\" (UniqueName: \"kubernetes.io/projected/d2a59049-7024-4284-9c8c-8cd039bab9ec-kube-api-access-r2wnv\") pod \"d2a59049-7024-4284-9c8c-8cd039bab9ec\" (UID: \"d2a59049-7024-4284-9c8c-8cd039bab9ec\") "
Jul 31 15:08:09.781823 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:09.781663    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2a59049-7024-4284-9c8c-8cd039bab9ec-tls-certs\") pod \"d2a59049-7024-4284-9c8c-8cd039bab9ec\" (UID: \"d2a59049-7024-4284-9c8c-8cd039bab9ec\") "
Jul 31 15:08:09.783736 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:09.783710    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a59049-7024-4284-9c8c-8cd039bab9ec-kube-api-access-r2wnv" (OuterVolumeSpecName: "kube-api-access-r2wnv") pod "d2a59049-7024-4284-9c8c-8cd039bab9ec" (UID: "d2a59049-7024-4284-9c8c-8cd039bab9ec"). InnerVolumeSpecName "kube-api-access-r2wnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:08:09.783881 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:09.783865    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a59049-7024-4284-9c8c-8cd039bab9ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2a59049-7024-4284-9c8c-8cd039bab9ec" (UID: "d2a59049-7024-4284-9c8c-8cd039bab9ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:08:09.882461 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:09.882425    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2wnv\" (UniqueName: \"kubernetes.io/projected/d2a59049-7024-4284-9c8c-8cd039bab9ec-kube-api-access-r2wnv\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:08:09.882461 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:09.882453    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2a59049-7024-4284-9c8c-8cd039bab9ec-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:08:10.165575 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:10.165436    2578 generic.go:356] "Generic (PLEG): container finished" podID="d2a59049-7024-4284-9c8c-8cd039bab9ec" containerID="82e9aecab4e9c248f80531299d5d3a95d3e0abb8407a414a96b65ac3c5cf5df7" exitCode=0
Jul 31 15:08:10.165575 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:10.165501    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"
Jul 31 15:08:10.165575 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:10.165538    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx" event={"ID":"d2a59049-7024-4284-9c8c-8cd039bab9ec","Type":"ContainerDied","Data":"82e9aecab4e9c248f80531299d5d3a95d3e0abb8407a414a96b65ac3c5cf5df7"}
Jul 31 15:08:10.165897 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:10.165625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx" event={"ID":"d2a59049-7024-4284-9c8c-8cd039bab9ec","Type":"ContainerDied","Data":"aa542a72d298e073283c1cf893cf38e5752655ded4e2cc14e86ca9ae021be8a7"}
Jul 31 15:08:10.165897 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:10.165666    2578 scope.go:117] "RemoveContainer" containerID="82e9aecab4e9c248f80531299d5d3a95d3e0abb8407a414a96b65ac3c5cf5df7"
Jul 31 15:08:10.180666 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:10.180553    2578 scope.go:117] "RemoveContainer" containerID="82e9aecab4e9c248f80531299d5d3a95d3e0abb8407a414a96b65ac3c5cf5df7"
Jul 31 15:08:10.181461 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:08:10.181332    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e9aecab4e9c248f80531299d5d3a95d3e0abb8407a414a96b65ac3c5cf5df7\": container with ID starting with 82e9aecab4e9c248f80531299d5d3a95d3e0abb8407a414a96b65ac3c5cf5df7 not found: ID does not exist" containerID="82e9aecab4e9c248f80531299d5d3a95d3e0abb8407a414a96b65ac3c5cf5df7"
Jul 31 15:08:10.181461 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:10.181449    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e9aecab4e9c248f80531299d5d3a95d3e0abb8407a414a96b65ac3c5cf5df7"} err="failed to get container status \"82e9aecab4e9c248f80531299d5d3a95d3e0abb8407a414a96b65ac3c5cf5df7\": rpc error: code = NotFound desc = could not find container \"82e9aecab4e9c248f80531299d5d3a95d3e0abb8407a414a96b65ac3c5cf5df7\": container with ID starting with 82e9aecab4e9c248f80531299d5d3a95d3e0abb8407a414a96b65ac3c5cf5df7 not found: ID does not exist"
Jul 31 15:08:10.196483 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:10.196445    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"]
Jul 31 15:08:10.201970 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:10.201943    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587296xx"]
Jul 31 15:08:10.490261 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:10.490226    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2a59049-7024-4284-9c8c-8cd039bab9ec" path="/var/lib/kubelet/pods/d2a59049-7024-4284-9c8c-8cd039bab9ec/volumes"
Jul 31 15:08:15.715327 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:15.715278    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 31 15:08:23.743802 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:23.743768    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"]
Jul 31 15:08:23.744374 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:23.744136    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2a59049-7024-4284-9c8c-8cd039bab9ec" containerName="main"
Jul 31 15:08:23.744374 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:23.744154    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a59049-7024-4284-9c8c-8cd039bab9ec" containerName="main"
Jul 31 15:08:23.744374 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:23.744234    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2a59049-7024-4284-9c8c-8cd039bab9ec" containerName="main"
Jul 31 15:08:23.747076 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:23.747053    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"
Jul 31 15:08:23.751037 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:23.751015    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 31 15:08:23.751151 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:23.751044    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 31 15:08:23.751215 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:23.751178    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-kc9xd\""
Jul 31 15:08:23.751295 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:23.751279    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 31 15:08:23.751405 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:23.751389    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-hmprv\""
Jul 31 15:08:23.760889 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:23.760866    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"]
Jul 31 15:08:23.796717 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:23.796679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b578e5d5-6049-4a1a-afd6-38c6d7c8374b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw\" (UID: \"b578e5d5-6049-4a1a-afd6-38c6d7c8374b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"
Jul 31 15:08:23.796895 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:23.796758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7h47\" (UniqueName: \"kubernetes.io/projected/b578e5d5-6049-4a1a-afd6-38c6d7c8374b-kube-api-access-x7h47\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw\" (UID: \"b578e5d5-6049-4a1a-afd6-38c6d7c8374b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"
Jul 31 15:08:23.898279 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:23.898208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7h47\" (UniqueName: \"kubernetes.io/projected/b578e5d5-6049-4a1a-afd6-38c6d7c8374b-kube-api-access-x7h47\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw\" (UID: \"b578e5d5-6049-4a1a-afd6-38c6d7c8374b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"
Jul 31 15:08:23.898482 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:23.898324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b578e5d5-6049-4a1a-afd6-38c6d7c8374b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw\" (UID: \"b578e5d5-6049-4a1a-afd6-38c6d7c8374b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"
Jul 31 15:08:23.900786 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:23.900761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b578e5d5-6049-4a1a-afd6-38c6d7c8374b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw\" (UID: \"b578e5d5-6049-4a1a-afd6-38c6d7c8374b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"
Jul 31 15:08:23.909241 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:23.909218    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7h47\" (UniqueName: \"kubernetes.io/projected/b578e5d5-6049-4a1a-afd6-38c6d7c8374b-kube-api-access-x7h47\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw\" (UID: \"b578e5d5-6049-4a1a-afd6-38c6d7c8374b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"
Jul 31 15:08:24.061575 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:24.061481    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"
Jul 31 15:08:24.202557 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:24.202532    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"]
Jul 31 15:08:24.205852 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:08:24.205820    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb578e5d5_6049_4a1a_afd6_38c6d7c8374b.slice/crio-ab4e9651a57efc95b3529bd8fda892948eb97dca1839762bcdda0cc97d3143a9 WatchSource:0}: Error finding container ab4e9651a57efc95b3529bd8fda892948eb97dca1839762bcdda0cc97d3143a9: Status 404 returned error can't find the container with id ab4e9651a57efc95b3529bd8fda892948eb97dca1839762bcdda0cc97d3143a9
Jul 31 15:08:24.222650 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:24.222622    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw" event={"ID":"b578e5d5-6049-4a1a-afd6-38c6d7c8374b","Type":"ContainerStarted","Data":"ab4e9651a57efc95b3529bd8fda892948eb97dca1839762bcdda0cc97d3143a9"}
Jul 31 15:08:25.228572 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:25.228536    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw" event={"ID":"b578e5d5-6049-4a1a-afd6-38c6d7c8374b","Type":"ContainerStarted","Data":"8526bac66f03d325fa9a8bc1a4c7c2349fdb28ab702cad55d4d836da6d85fe0c"}
Jul 31 15:08:25.228977 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:25.228654    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"
Jul 31 15:08:25.257848 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:25.257795    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw" podStartSLOduration=2.257776986 podStartE2EDuration="2.257776986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:08:25.25592544 +0000 UTC m=+1203.265137308" watchObservedRunningTime="2026-07-31 15:08:25.257776986 +0000 UTC m=+1203.266988858"
Jul 31 15:08:25.715751 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:25.715649    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 31 15:08:35.716152 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:35.716107    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 31 15:08:45.715700 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:45.715642    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 31 15:08:55.725692 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:55.725660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:08:55.733252 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:55.733225    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:08:56.235006 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:56.234976    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"
Jul 31 15:08:57.575890 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:57.575855    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"]
Jul 31 15:08:58.846779 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:08:58.846749    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"]
Jul 31 15:09:06.234890 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:06.234849    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw" podUID="b578e5d5-6049-4a1a-afd6-38c6d7c8374b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 15:09:10.533603 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:10.533567    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw" message="" err="command '/bin/sleep 15' exited with 137: "
Jul 31 15:09:10.533901 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:09:10.533608    2578 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw" podUID="b578e5d5-6049-4a1a-afd6-38c6d7c8374b" containerName="main" containerID="cri-o://8526bac66f03d325fa9a8bc1a4c7c2349fdb28ab702cad55d4d836da6d85fe0c"
Jul 31 15:09:10.533901 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:10.533652    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw" podUID="b578e5d5-6049-4a1a-afd6-38c6d7c8374b" containerName="main" containerID="cri-o://8526bac66f03d325fa9a8bc1a4c7c2349fdb28ab702cad55d4d836da6d85fe0c" gracePeriod=49
Jul 31 15:09:10.676254 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:10.676224    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw_b578e5d5-6049-4a1a-afd6-38c6d7c8374b/main/0.log"
Jul 31 15:09:10.680726 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:10.680700    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"
Jul 31 15:09:10.784029 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:10.783954    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7h47\" (UniqueName: \"kubernetes.io/projected/b578e5d5-6049-4a1a-afd6-38c6d7c8374b-kube-api-access-x7h47\") pod \"b578e5d5-6049-4a1a-afd6-38c6d7c8374b\" (UID: \"b578e5d5-6049-4a1a-afd6-38c6d7c8374b\") "
Jul 31 15:09:10.784029 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:10.784019    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b578e5d5-6049-4a1a-afd6-38c6d7c8374b-tls-certs\") pod \"b578e5d5-6049-4a1a-afd6-38c6d7c8374b\" (UID: \"b578e5d5-6049-4a1a-afd6-38c6d7c8374b\") "
Jul 31 15:09:10.785991 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:10.785970    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b578e5d5-6049-4a1a-afd6-38c6d7c8374b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b578e5d5-6049-4a1a-afd6-38c6d7c8374b" (UID: "b578e5d5-6049-4a1a-afd6-38c6d7c8374b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:09:10.786090 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:10.786073    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b578e5d5-6049-4a1a-afd6-38c6d7c8374b-kube-api-access-x7h47" (OuterVolumeSpecName: "kube-api-access-x7h47") pod "b578e5d5-6049-4a1a-afd6-38c6d7c8374b" (UID: "b578e5d5-6049-4a1a-afd6-38c6d7c8374b"). InnerVolumeSpecName "kube-api-access-x7h47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:09:10.884894 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:10.884843    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7h47\" (UniqueName: \"kubernetes.io/projected/b578e5d5-6049-4a1a-afd6-38c6d7c8374b-kube-api-access-x7h47\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:09:10.884894 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:10.884888    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b578e5d5-6049-4a1a-afd6-38c6d7c8374b-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:09:11.415998 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:11.415970    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw_b578e5d5-6049-4a1a-afd6-38c6d7c8374b/main/0.log"
Jul 31 15:09:11.416177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:11.416012    2578 generic.go:356] "Generic (PLEG): container finished" podID="b578e5d5-6049-4a1a-afd6-38c6d7c8374b" containerID="8526bac66f03d325fa9a8bc1a4c7c2349fdb28ab702cad55d4d836da6d85fe0c" exitCode=1
Jul 31 15:09:11.416177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:11.416071    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"
Jul 31 15:09:11.416177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:11.416111    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw" event={"ID":"b578e5d5-6049-4a1a-afd6-38c6d7c8374b","Type":"ContainerDied","Data":"8526bac66f03d325fa9a8bc1a4c7c2349fdb28ab702cad55d4d836da6d85fe0c"}
Jul 31 15:09:11.416280 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:11.416188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw" event={"ID":"b578e5d5-6049-4a1a-afd6-38c6d7c8374b","Type":"ContainerDied","Data":"ab4e9651a57efc95b3529bd8fda892948eb97dca1839762bcdda0cc97d3143a9"}
Jul 31 15:09:11.416280 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:11.416213    2578 scope.go:117] "RemoveContainer" containerID="8526bac66f03d325fa9a8bc1a4c7c2349fdb28ab702cad55d4d836da6d85fe0c"
Jul 31 15:09:11.425004 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:11.424978    2578 scope.go:117] "RemoveContainer" containerID="8526bac66f03d325fa9a8bc1a4c7c2349fdb28ab702cad55d4d836da6d85fe0c"
Jul 31 15:09:11.425255 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:09:11.425239    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8526bac66f03d325fa9a8bc1a4c7c2349fdb28ab702cad55d4d836da6d85fe0c\": container with ID starting with 8526bac66f03d325fa9a8bc1a4c7c2349fdb28ab702cad55d4d836da6d85fe0c not found: ID does not exist" containerID="8526bac66f03d325fa9a8bc1a4c7c2349fdb28ab702cad55d4d836da6d85fe0c"
Jul 31 15:09:11.425304 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:11.425265    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8526bac66f03d325fa9a8bc1a4c7c2349fdb28ab702cad55d4d836da6d85fe0c"} err="failed to get container status \"8526bac66f03d325fa9a8bc1a4c7c2349fdb28ab702cad55d4d836da6d85fe0c\": rpc error: code = NotFound desc = could not find container \"8526bac66f03d325fa9a8bc1a4c7c2349fdb28ab702cad55d4d836da6d85fe0c\": container with ID starting with 8526bac66f03d325fa9a8bc1a4c7c2349fdb28ab702cad55d4d836da6d85fe0c not found: ID does not exist"
Jul 31 15:09:11.437834 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:11.437811    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"]
Jul 31 15:09:11.442251 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:11.442229    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfqdtxw"]
Jul 31 15:09:12.489090 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:12.489059    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b578e5d5-6049-4a1a-afd6-38c6d7c8374b" path="/var/lib/kubelet/pods/b578e5d5-6049-4a1a-afd6-38c6d7c8374b/volumes"
Jul 31 15:09:12.597577 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:12.597536    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="main" containerID="cri-o://adbac8f524192a6243302d33401cea2ca56dec5201a271f5ed88390e934c6952" gracePeriod=45
Jul 31 15:09:12.726238 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:12.726203    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
Jul 31 15:09:13.251516 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.251493    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:09:13.303045 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.303013    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27cb2126-dc8c-4659-9215-4f5a0ff9947e-cabundle-cert\") pod \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") "
Jul 31 15:09:13.303209 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.303057    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-tmp-dir\") pod \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") "
Jul 31 15:09:13.303209 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.303084    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf4bf\" (UniqueName: \"kubernetes.io/projected/27cb2126-dc8c-4659-9215-4f5a0ff9947e-kube-api-access-tf4bf\") pod \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") "
Jul 31 15:09:13.303209 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.303117    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-dshm\") pod \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") "
Jul 31 15:09:13.303209 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.303135    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-model-cache\") pod \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") "
Jul 31 15:09:13.303209 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.303155    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27cb2126-dc8c-4659-9215-4f5a0ff9947e-tls-certs\") pod \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") "
Jul 31 15:09:13.303209 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.303182    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-kserve-provision-location\") pod \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") "
Jul 31 15:09:13.303515 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.303260    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-home\") pod \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\" (UID: \"27cb2126-dc8c-4659-9215-4f5a0ff9947e\") "
Jul 31 15:09:13.303574 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.303527    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-model-cache" (OuterVolumeSpecName: "model-cache") pod "27cb2126-dc8c-4659-9215-4f5a0ff9947e" (UID: "27cb2126-dc8c-4659-9215-4f5a0ff9947e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:09:13.303668 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.303622    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27cb2126-dc8c-4659-9215-4f5a0ff9947e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "27cb2126-dc8c-4659-9215-4f5a0ff9947e" (UID: "27cb2126-dc8c-4659-9215-4f5a0ff9947e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:09:13.304116 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.304075    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-home" (OuterVolumeSpecName: "home") pod "27cb2126-dc8c-4659-9215-4f5a0ff9947e" (UID: "27cb2126-dc8c-4659-9215-4f5a0ff9947e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:09:13.305652 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.305627    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27cb2126-dc8c-4659-9215-4f5a0ff9947e-kube-api-access-tf4bf" (OuterVolumeSpecName: "kube-api-access-tf4bf") pod "27cb2126-dc8c-4659-9215-4f5a0ff9947e" (UID: "27cb2126-dc8c-4659-9215-4f5a0ff9947e"). InnerVolumeSpecName "kube-api-access-tf4bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:09:13.305761 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.305627    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-dshm" (OuterVolumeSpecName: "dshm") pod "27cb2126-dc8c-4659-9215-4f5a0ff9947e" (UID: "27cb2126-dc8c-4659-9215-4f5a0ff9947e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:09:13.305761 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.305722    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27cb2126-dc8c-4659-9215-4f5a0ff9947e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27cb2126-dc8c-4659-9215-4f5a0ff9947e" (UID: "27cb2126-dc8c-4659-9215-4f5a0ff9947e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:09:13.316113 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.316081    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27cb2126-dc8c-4659-9215-4f5a0ff9947e" (UID: "27cb2126-dc8c-4659-9215-4f5a0ff9947e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:09:13.327439 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.327415    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27cb2126-dc8c-4659-9215-4f5a0ff9947e" (UID: "27cb2126-dc8c-4659-9215-4f5a0ff9947e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:09:13.404435 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.404332    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:09:13.404435 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.404386    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27cb2126-dc8c-4659-9215-4f5a0ff9947e-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:09:13.404435 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.404398    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:09:13.404435 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.404407    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf4bf\" (UniqueName: \"kubernetes.io/projected/27cb2126-dc8c-4659-9215-4f5a0ff9947e-kube-api-access-tf4bf\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:09:13.404435 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.404415    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:09:13.404435 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.404422    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:09:13.404435 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.404430    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27cb2126-dc8c-4659-9215-4f5a0ff9947e-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:09:13.404435 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.404438    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27cb2126-dc8c-4659-9215-4f5a0ff9947e-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:09:13.424998 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.424971    2578 generic.go:356] "Generic (PLEG): container finished" podID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerID="adbac8f524192a6243302d33401cea2ca56dec5201a271f5ed88390e934c6952" exitCode=0
Jul 31 15:09:13.425103 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.425019    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"
Jul 31 15:09:13.425233 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.425021    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" event={"ID":"27cb2126-dc8c-4659-9215-4f5a0ff9947e","Type":"ContainerDied","Data":"adbac8f524192a6243302d33401cea2ca56dec5201a271f5ed88390e934c6952"}
Jul 31 15:09:13.425233 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.425169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr" event={"ID":"27cb2126-dc8c-4659-9215-4f5a0ff9947e","Type":"ContainerDied","Data":"0c083c52e8b12d35f8273526d3a996671c23d21af3099b4dfaec77b558e75d05"}
Jul 31 15:09:13.425233 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.425216    2578 scope.go:117] "RemoveContainer" containerID="adbac8f524192a6243302d33401cea2ca56dec5201a271f5ed88390e934c6952"
Jul 31 15:09:13.433680 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.433664    2578 scope.go:117] "RemoveContainer" containerID="457f25fba6f4d2057a66929fb35153e9d62f5dff93147eb31da0a65a8dfa222b"
Jul 31 15:09:13.440929 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.440911    2578 scope.go:117] "RemoveContainer" containerID="adbac8f524192a6243302d33401cea2ca56dec5201a271f5ed88390e934c6952"
Jul 31 15:09:13.441183 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:09:13.441163    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adbac8f524192a6243302d33401cea2ca56dec5201a271f5ed88390e934c6952\": container with ID starting with adbac8f524192a6243302d33401cea2ca56dec5201a271f5ed88390e934c6952 not found: ID does not exist" containerID="adbac8f524192a6243302d33401cea2ca56dec5201a271f5ed88390e934c6952"
Jul 31 15:09:13.441226 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.441192    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adbac8f524192a6243302d33401cea2ca56dec5201a271f5ed88390e934c6952"} err="failed to get container status \"adbac8f524192a6243302d33401cea2ca56dec5201a271f5ed88390e934c6952\": rpc error: code = NotFound desc = could not find container \"adbac8f524192a6243302d33401cea2ca56dec5201a271f5ed88390e934c6952\": container with ID starting with adbac8f524192a6243302d33401cea2ca56dec5201a271f5ed88390e934c6952 not found: ID does not exist"
Jul 31 15:09:13.441226 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.441209    2578 scope.go:117] "RemoveContainer" containerID="457f25fba6f4d2057a66929fb35153e9d62f5dff93147eb31da0a65a8dfa222b"
Jul 31 15:09:13.441469 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:09:13.441449    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457f25fba6f4d2057a66929fb35153e9d62f5dff93147eb31da0a65a8dfa222b\": container with ID starting with 457f25fba6f4d2057a66929fb35153e9d62f5dff93147eb31da0a65a8dfa222b not found: ID does not exist" containerID="457f25fba6f4d2057a66929fb35153e9d62f5dff93147eb31da0a65a8dfa222b"
Jul 31 15:09:13.441513 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.441477    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457f25fba6f4d2057a66929fb35153e9d62f5dff93147eb31da0a65a8dfa222b"} err="failed to get container status \"457f25fba6f4d2057a66929fb35153e9d62f5dff93147eb31da0a65a8dfa222b\": rpc error: code = NotFound desc = could not find container \"457f25fba6f4d2057a66929fb35153e9d62f5dff93147eb31da0a65a8dfa222b\": container with ID starting with 457f25fba6f4d2057a66929fb35153e9d62f5dff93147eb31da0a65a8dfa222b not found: ID does not exist"
Jul 31 15:09:13.448067 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.448045    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"]
Jul 31 15:09:13.454739 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:13.454716    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ffd754788-qnrjr"]
Jul 31 15:09:14.489112 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:14.489036    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" path="/var/lib/kubelet/pods/27cb2126-dc8c-4659-9215-4f5a0ff9947e/volumes"
Jul 31 15:09:25.595912 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.595881    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"]
Jul 31 15:09:25.596451 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.596146    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="storage-initializer"
Jul 31 15:09:25.596451 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.596157    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="storage-initializer"
Jul 31 15:09:25.596451 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.596169    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b578e5d5-6049-4a1a-afd6-38c6d7c8374b" containerName="main"
Jul 31 15:09:25.596451 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.596174    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b578e5d5-6049-4a1a-afd6-38c6d7c8374b" containerName="main"
Jul 31 15:09:25.596451 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.596181    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="main"
Jul 31 15:09:25.596451 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.596187    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="main"
Jul 31 15:09:25.596451 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.596227    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b578e5d5-6049-4a1a-afd6-38c6d7c8374b" containerName="main"
Jul 31 15:09:25.596451 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.596236    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="27cb2126-dc8c-4659-9215-4f5a0ff9947e" containerName="main"
Jul 31 15:09:25.600286 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.600262    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.604863 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.604837    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:09:25.605037 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.605019    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Jul 31 15:09:25.605151 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.605130    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Jul 31 15:09:25.605257 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.605241    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 31 15:09:25.605382 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.605366    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Jul 31 15:09:25.606125 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.606108    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-w8lr9\""
Jul 31 15:09:25.612740 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.612719    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"]
Jul 31 15:09:25.694006 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.693971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5837f3f2-5394-4d2b-92df-f7d9619698c9-tls-certs\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.694006 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.694008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-dshm\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.694247 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.694033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5837f3f2-5394-4d2b-92df-f7d9619698c9-cabundle-cert\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.694247 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.694054    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6fft\" (UniqueName: \"kubernetes.io/projected/5837f3f2-5394-4d2b-92df-f7d9619698c9-kube-api-access-f6fft\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.694247 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.694118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-tmp-dir\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.694247 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.694147    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-model-cache\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.694247 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.694175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-home\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.694471 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.694243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-kserve-provision-location\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.795249 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.795215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5837f3f2-5394-4d2b-92df-f7d9619698c9-tls-certs\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.795249 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.795249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-dshm\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.795555 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.795271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5837f3f2-5394-4d2b-92df-f7d9619698c9-cabundle-cert\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.795555 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.795294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6fft\" (UniqueName: \"kubernetes.io/projected/5837f3f2-5394-4d2b-92df-f7d9619698c9-kube-api-access-f6fft\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.795555 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.795322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-tmp-dir\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.795555 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.795372    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-model-cache\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.795555 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.795504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-home\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.795555 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.795552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-kserve-provision-location\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.795866 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.795726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-tmp-dir\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.795928 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.795869    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-home\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.795928 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.795875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-model-cache\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.795928 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.795908    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-kserve-provision-location\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.796192 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.796171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5837f3f2-5394-4d2b-92df-f7d9619698c9-cabundle-cert\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.797536 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.797514    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-dshm\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.797937 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.797918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5837f3f2-5394-4d2b-92df-f7d9619698c9-tls-certs\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.807078 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.807057    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6fft\" (UniqueName: \"kubernetes.io/projected/5837f3f2-5394-4d2b-92df-f7d9619698c9-kube-api-access-f6fft\") pod \"auth-disabled-test-kserve-6996464c95-p4t87\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:25.910409 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:25.910309    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:26.045578 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:26.045553    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"]
Jul 31 15:09:26.047669 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:09:26.047637    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5837f3f2_5394_4d2b_92df_f7d9619698c9.slice/crio-20122258c896bc63e2b0691d56accd243a0abda30571fbc9c8974f21166b837a WatchSource:0}: Error finding container 20122258c896bc63e2b0691d56accd243a0abda30571fbc9c8974f21166b837a: Status 404 returned error can't find the container with id 20122258c896bc63e2b0691d56accd243a0abda30571fbc9c8974f21166b837a
Jul 31 15:09:26.477913 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:26.477877    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" event={"ID":"5837f3f2-5394-4d2b-92df-f7d9619698c9","Type":"ContainerStarted","Data":"50779e3949be6a464ac68bd78f8ac72149bda2fc956157d1f46d6257affe1320"}
Jul 31 15:09:26.478077 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:26.477942    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" event={"ID":"5837f3f2-5394-4d2b-92df-f7d9619698c9","Type":"ContainerStarted","Data":"20122258c896bc63e2b0691d56accd243a0abda30571fbc9c8974f21166b837a"}
Jul 31 15:09:28.492465 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:28.492410    2578 generic.go:356] "Generic (PLEG): container finished" podID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerID="50779e3949be6a464ac68bd78f8ac72149bda2fc956157d1f46d6257affe1320" exitCode=0
Jul 31 15:09:28.493521 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:28.493488    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" event={"ID":"5837f3f2-5394-4d2b-92df-f7d9619698c9","Type":"ContainerDied","Data":"50779e3949be6a464ac68bd78f8ac72149bda2fc956157d1f46d6257affe1320"}
Jul 31 15:09:29.498657 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:29.498623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" event={"ID":"5837f3f2-5394-4d2b-92df-f7d9619698c9","Type":"ContainerStarted","Data":"13188f8d2185303431c94ab61896f16a82cb3d43539e87ce2dbb6f2042491fc4"}
Jul 31 15:09:29.524139 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:29.524070    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" podStartSLOduration=2.351001692 podStartE2EDuration="4.52404852s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.173046828s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:09:29.521720279 +0000 UTC m=+1267.530932148" watchObservedRunningTime="2026-07-31 15:09:29.52404852 +0000 UTC m=+1267.533260389"
Jul 31 15:09:29.911133 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:29.911051    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:35.911330 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:35.911273    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:09:35.912855 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:35.912825    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 31 15:09:40.370196 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.370162    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"]
Jul 31 15:09:40.372402 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.372384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:09:40.376796 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.376754    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-wdhnl\""
Jul 31 15:09:40.376796 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.376806    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-7f8hj\""
Jul 31 15:09:40.377014 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.376829    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Jul 31 15:09:40.377014 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.376868    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 31 15:09:40.377014 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.376914    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Jul 31 15:09:40.381502 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.381481    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"]
Jul 31 15:09:40.518580 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.518541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcsmr\" (UniqueName: \"kubernetes.io/projected/198df827-cb20-4ff6-a5b8-52b492ff1eb6-kube-api-access-dcsmr\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795jmxv6\" (UID: \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:09:40.518580 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.518583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/198df827-cb20-4ff6-a5b8-52b492ff1eb6-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795jmxv6\" (UID: \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:09:40.518820 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.518658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/198df827-cb20-4ff6-a5b8-52b492ff1eb6-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795jmxv6\" (UID: \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:09:40.518820 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.518712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198df827-cb20-4ff6-a5b8-52b492ff1eb6-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795jmxv6\" (UID: \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:09:40.619541 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.619498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/198df827-cb20-4ff6-a5b8-52b492ff1eb6-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795jmxv6\" (UID: \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:09:40.619739 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.619550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198df827-cb20-4ff6-a5b8-52b492ff1eb6-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795jmxv6\" (UID: \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:09:40.619739 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.619713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcsmr\" (UniqueName: \"kubernetes.io/projected/198df827-cb20-4ff6-a5b8-52b492ff1eb6-kube-api-access-dcsmr\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795jmxv6\" (UID: \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:09:40.619864 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.619755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/198df827-cb20-4ff6-a5b8-52b492ff1eb6-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795jmxv6\" (UID: \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:09:40.619965 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.619941    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/198df827-cb20-4ff6-a5b8-52b492ff1eb6-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795jmxv6\" (UID: \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:09:40.620141 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.620120    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/198df827-cb20-4ff6-a5b8-52b492ff1eb6-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795jmxv6\" (UID: \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:09:40.622050 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.622003    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198df827-cb20-4ff6-a5b8-52b492ff1eb6-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795jmxv6\" (UID: \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:09:40.629693 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.629671    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcsmr\" (UniqueName: \"kubernetes.io/projected/198df827-cb20-4ff6-a5b8-52b492ff1eb6-kube-api-access-dcsmr\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795jmxv6\" (UID: \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:09:40.683667 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.683629    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:09:40.816416 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:40.816382    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"]
Jul 31 15:09:40.819735 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:09:40.819709    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198df827_cb20_4ff6_a5b8_52b492ff1eb6.slice/crio-c5a8fc221a9786d8f07e9ca6c851f9529e2372a650d1fea47d31c7e0fc06172f WatchSource:0}: Error finding container c5a8fc221a9786d8f07e9ca6c851f9529e2372a650d1fea47d31c7e0fc06172f: Status 404 returned error can't find the container with id c5a8fc221a9786d8f07e9ca6c851f9529e2372a650d1fea47d31c7e0fc06172f
Jul 31 15:09:41.547125 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:41.547090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6" event={"ID":"198df827-cb20-4ff6-a5b8-52b492ff1eb6","Type":"ContainerStarted","Data":"3b5972ae6f77416efb20b85e774ea17e78af80211a30415a1b489731630d6075"}
Jul 31 15:09:41.547525 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:41.547172    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6" event={"ID":"198df827-cb20-4ff6-a5b8-52b492ff1eb6","Type":"ContainerStarted","Data":"c5a8fc221a9786d8f07e9ca6c851f9529e2372a650d1fea47d31c7e0fc06172f"}
Jul 31 15:09:41.547525 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:41.547201    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:09:41.573545 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:41.573495    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6" podStartSLOduration=1.573479747 podStartE2EDuration="1.573479747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:09:41.572253376 +0000 UTC m=+1279.581465247" watchObservedRunningTime="2026-07-31 15:09:41.573479747 +0000 UTC m=+1279.582691616"
Jul 31 15:09:45.911139 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:45.911092    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 31 15:09:55.911902 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:09:55.911845    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 31 15:10:05.911282 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:05.911226    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 31 15:10:12.552302 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:12.552270    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:10:14.233525 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:14.233490    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"]
Jul 31 15:10:15.911302 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:15.911249    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 31 15:10:22.553226 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:22.553187    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6" podUID="198df827-cb20-4ff6-a5b8-52b492ff1eb6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 15:10:25.911312 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:25.911261    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 31 15:10:29.252669 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.252612    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6" podUID="198df827-cb20-4ff6-a5b8-52b492ff1eb6" containerName="main" containerID="cri-o://3b5972ae6f77416efb20b85e774ea17e78af80211a30415a1b489731630d6075" gracePeriod=45
Jul 31 15:10:29.500106 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.500082    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:10:29.627164 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.627079    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcsmr\" (UniqueName: \"kubernetes.io/projected/198df827-cb20-4ff6-a5b8-52b492ff1eb6-kube-api-access-dcsmr\") pod \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\" (UID: \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\") "
Jul 31 15:10:29.627164 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.627131    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198df827-cb20-4ff6-a5b8-52b492ff1eb6-tls-certs\") pod \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\" (UID: \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\") "
Jul 31 15:10:29.627438 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.627175    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/198df827-cb20-4ff6-a5b8-52b492ff1eb6-cachi2-cache\") pod \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\" (UID: \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\") "
Jul 31 15:10:29.627438 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.627234    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/198df827-cb20-4ff6-a5b8-52b492ff1eb6-tokenizer-cache\") pod \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\" (UID: \"198df827-cb20-4ff6-a5b8-52b492ff1eb6\") "
Jul 31 15:10:29.627571 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.627542    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198df827-cb20-4ff6-a5b8-52b492ff1eb6-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "198df827-cb20-4ff6-a5b8-52b492ff1eb6" (UID: "198df827-cb20-4ff6-a5b8-52b492ff1eb6"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:10:29.627632 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.627574    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198df827-cb20-4ff6-a5b8-52b492ff1eb6-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "198df827-cb20-4ff6-a5b8-52b492ff1eb6" (UID: "198df827-cb20-4ff6-a5b8-52b492ff1eb6"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:10:29.629402 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.629314    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198df827-cb20-4ff6-a5b8-52b492ff1eb6-kube-api-access-dcsmr" (OuterVolumeSpecName: "kube-api-access-dcsmr") pod "198df827-cb20-4ff6-a5b8-52b492ff1eb6" (UID: "198df827-cb20-4ff6-a5b8-52b492ff1eb6"). InnerVolumeSpecName "kube-api-access-dcsmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:10:29.629402 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.629397    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/198df827-cb20-4ff6-a5b8-52b492ff1eb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "198df827-cb20-4ff6-a5b8-52b492ff1eb6" (UID: "198df827-cb20-4ff6-a5b8-52b492ff1eb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:10:29.728134 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.728086    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcsmr\" (UniqueName: \"kubernetes.io/projected/198df827-cb20-4ff6-a5b8-52b492ff1eb6-kube-api-access-dcsmr\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:10:29.728134 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.728125    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198df827-cb20-4ff6-a5b8-52b492ff1eb6-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:10:29.728134 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.728137    2578 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/198df827-cb20-4ff6-a5b8-52b492ff1eb6-cachi2-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:10:29.728134 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.728145    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/198df827-cb20-4ff6-a5b8-52b492ff1eb6-tokenizer-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:10:29.741077 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.741038    2578 generic.go:356] "Generic (PLEG): container finished" podID="198df827-cb20-4ff6-a5b8-52b492ff1eb6" containerID="3b5972ae6f77416efb20b85e774ea17e78af80211a30415a1b489731630d6075" exitCode=0
Jul 31 15:10:29.741285 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.741081    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"
Jul 31 15:10:29.741285 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.741088    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6" event={"ID":"198df827-cb20-4ff6-a5b8-52b492ff1eb6","Type":"ContainerDied","Data":"3b5972ae6f77416efb20b85e774ea17e78af80211a30415a1b489731630d6075"}
Jul 31 15:10:29.741285 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.741214    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6" event={"ID":"198df827-cb20-4ff6-a5b8-52b492ff1eb6","Type":"ContainerDied","Data":"c5a8fc221a9786d8f07e9ca6c851f9529e2372a650d1fea47d31c7e0fc06172f"}
Jul 31 15:10:29.741285 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.741246    2578 scope.go:117] "RemoveContainer" containerID="3b5972ae6f77416efb20b85e774ea17e78af80211a30415a1b489731630d6075"
Jul 31 15:10:29.750398 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.750380    2578 scope.go:117] "RemoveContainer" containerID="3b5972ae6f77416efb20b85e774ea17e78af80211a30415a1b489731630d6075"
Jul 31 15:10:29.750676 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:10:29.750658    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5972ae6f77416efb20b85e774ea17e78af80211a30415a1b489731630d6075\": container with ID starting with 3b5972ae6f77416efb20b85e774ea17e78af80211a30415a1b489731630d6075 not found: ID does not exist" containerID="3b5972ae6f77416efb20b85e774ea17e78af80211a30415a1b489731630d6075"
Jul 31 15:10:29.750738 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.750688    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5972ae6f77416efb20b85e774ea17e78af80211a30415a1b489731630d6075"} err="failed to get container status \"3b5972ae6f77416efb20b85e774ea17e78af80211a30415a1b489731630d6075\": rpc error: code = NotFound desc = could not find container \"3b5972ae6f77416efb20b85e774ea17e78af80211a30415a1b489731630d6075\": container with ID starting with 3b5972ae6f77416efb20b85e774ea17e78af80211a30415a1b489731630d6075 not found: ID does not exist"
Jul 31 15:10:29.766001 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.765969    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"]
Jul 31 15:10:29.772210 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:29.772182    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795jmxv6"]
Jul 31 15:10:30.489636 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:30.489599    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="198df827-cb20-4ff6-a5b8-52b492ff1eb6" path="/var/lib/kubelet/pods/198df827-cb20-4ff6-a5b8-52b492ff1eb6/volumes"
Jul 31 15:10:35.911546 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:35.911498    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 31 15:10:38.359970 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.359935    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"]
Jul 31 15:10:38.360434 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.360412    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="198df827-cb20-4ff6-a5b8-52b492ff1eb6" containerName="main"
Jul 31 15:10:38.360485 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.360438    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="198df827-cb20-4ff6-a5b8-52b492ff1eb6" containerName="main"
Jul 31 15:10:38.360532 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.360521    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="198df827-cb20-4ff6-a5b8-52b492ff1eb6" containerName="main"
Jul 31 15:10:38.363400 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.363382    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.368862 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.368837    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 31 15:10:38.368993 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.368843    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-pt24g\""
Jul 31 15:10:38.368993 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.368843    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 31 15:10:38.369122 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.369011    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 31 15:10:38.377121 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.377092    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"]
Jul 31 15:10:38.500617 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.500583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.500806 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.500632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtsqb\" (UniqueName: \"kubernetes.io/projected/c63816e9-8042-4842-9d61-8817a972c480-kube-api-access-rtsqb\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.500806 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.500732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.500806 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.500770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.500806 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.500797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63816e9-8042-4842-9d61-8817a972c480-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.500980 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.500822    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.588042 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.588010    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"]
Jul 31 15:10:38.590240 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.590219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"
Jul 31 15:10:38.592843 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.592825    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-7tdkt\""
Jul 31 15:10:38.602106 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.602079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtsqb\" (UniqueName: \"kubernetes.io/projected/c63816e9-8042-4842-9d61-8817a972c480-kube-api-access-rtsqb\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.602225 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.602141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.602225 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.602170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.602225 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.602193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63816e9-8042-4842-9d61-8817a972c480-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.602225 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.602222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.602476 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.602270    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.602695 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.602647    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.602810 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.602710    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.602951 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.602927    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.604126 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.604096    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"]
Jul 31 15:10:38.605092 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.605063    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63816e9-8042-4842-9d61-8817a972c480-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.605203 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.605184    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.613396 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.613301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtsqb\" (UniqueName: \"kubernetes.io/projected/c63816e9-8042-4842-9d61-8817a972c480-kube-api-access-rtsqb\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.673036 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.672993    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:38.703649 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.703616    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3eba23d-9c53-4c44-91ca-b7f14e584423-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w\" (UID: \"d3eba23d-9c53-4c44-91ca-b7f14e584423\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"
Jul 31 15:10:38.703817 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.703791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdk7k\" (UniqueName: \"kubernetes.io/projected/d3eba23d-9c53-4c44-91ca-b7f14e584423-kube-api-access-hdk7k\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w\" (UID: \"d3eba23d-9c53-4c44-91ca-b7f14e584423\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"
Jul 31 15:10:38.805169 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.805135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdk7k\" (UniqueName: \"kubernetes.io/projected/d3eba23d-9c53-4c44-91ca-b7f14e584423-kube-api-access-hdk7k\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w\" (UID: \"d3eba23d-9c53-4c44-91ca-b7f14e584423\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"
Jul 31 15:10:38.805359 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.805186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3eba23d-9c53-4c44-91ca-b7f14e584423-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w\" (UID: \"d3eba23d-9c53-4c44-91ca-b7f14e584423\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"
Jul 31 15:10:38.807465 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.807444    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3eba23d-9c53-4c44-91ca-b7f14e584423-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w\" (UID: \"d3eba23d-9c53-4c44-91ca-b7f14e584423\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"
Jul 31 15:10:38.820974 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.816343    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"]
Jul 31 15:10:38.820974 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.818030    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdk7k\" (UniqueName: \"kubernetes.io/projected/d3eba23d-9c53-4c44-91ca-b7f14e584423-kube-api-access-hdk7k\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w\" (UID: \"d3eba23d-9c53-4c44-91ca-b7f14e584423\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"
Jul 31 15:10:38.822086 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:10:38.822053    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63816e9_8042_4842_9d61_8817a972c480.slice/crio-e645fe00193ae90a396e9af155e8f0c30463d7013953ea824ec725358a64a613 WatchSource:0}: Error finding container e645fe00193ae90a396e9af155e8f0c30463d7013953ea824ec725358a64a613: Status 404 returned error can't find the container with id e645fe00193ae90a396e9af155e8f0c30463d7013953ea824ec725358a64a613
Jul 31 15:10:38.900762 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:38.900726    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"
Jul 31 15:10:39.030309 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:39.030276    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"]
Jul 31 15:10:39.031624 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:10:39.031587    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3eba23d_9c53_4c44_91ca_b7f14e584423.slice/crio-f6eb7d729d95672ebac51e99dbbf73aaadd7b5f80dce57e24701f9db8826f05b WatchSource:0}: Error finding container f6eb7d729d95672ebac51e99dbbf73aaadd7b5f80dce57e24701f9db8826f05b: Status 404 returned error can't find the container with id f6eb7d729d95672ebac51e99dbbf73aaadd7b5f80dce57e24701f9db8826f05b
Jul 31 15:10:39.785404 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:39.785365    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8" event={"ID":"c63816e9-8042-4842-9d61-8817a972c480","Type":"ContainerStarted","Data":"0d40b2d50819e50a37355d579660a7c5de569039cdb1ecf68b02d5758ef54a8a"}
Jul 31 15:10:39.785838 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:39.785424    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8" event={"ID":"c63816e9-8042-4842-9d61-8817a972c480","Type":"ContainerStarted","Data":"e645fe00193ae90a396e9af155e8f0c30463d7013953ea824ec725358a64a613"}
Jul 31 15:10:39.786902 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:39.786882    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w" event={"ID":"d3eba23d-9c53-4c44-91ca-b7f14e584423","Type":"ContainerStarted","Data":"b16c265d7ac96e805157aaf04e1c13ba3f211316459aeb5a725d1b3ce58e0b8b"}
Jul 31 15:10:39.786963 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:39.786918    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w" event={"ID":"d3eba23d-9c53-4c44-91ca-b7f14e584423","Type":"ContainerStarted","Data":"f6eb7d729d95672ebac51e99dbbf73aaadd7b5f80dce57e24701f9db8826f05b"}
Jul 31 15:10:39.787000 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:39.786955    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"
Jul 31 15:10:39.807210 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:39.807159    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8" podStartSLOduration=1.807145258 podStartE2EDuration="1.807145258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:10:39.806062235 +0000 UTC m=+1337.815274105" watchObservedRunningTime="2026-07-31 15:10:39.807145258 +0000 UTC m=+1337.816357127"
Jul 31 15:10:39.825534 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:39.825484    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w" podStartSLOduration=1.8254670750000002 podStartE2EDuration="1.825467075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:10:39.823647704 +0000 UTC m=+1337.832859573" watchObservedRunningTime="2026-07-31 15:10:39.825467075 +0000 UTC m=+1337.834678947"
Jul 31 15:10:40.674052 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:40.674006    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:45.911287 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:45.911191    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 31 15:10:48.673614 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:48.673574    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:48.686266 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:48.686233    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:48.833046 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:48.833016    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:10:55.911582 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:10:55.911542    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 31 15:11:05.911334 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:05.911272    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 31 15:11:10.793856 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:10.793823    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"
Jul 31 15:11:12.473563 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:12.473522    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"]
Jul 31 15:11:12.490460 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:11:12.490147    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:11:12.490460 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:11:12.490460 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:11:12.490460 ip-10-0-133-32 kubenswrapper[2578]:  > containerID="0d40b2d50819e50a37355d579660a7c5de569039cdb1ecf68b02d5758ef54a8a" cmd=["/bin/sleep","15"]
Jul 31 15:11:12.490460 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:12.490213    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8" message="" err=<
Jul 31 15:11:12.490460 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:11:12.490460 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:11:12.490460 ip-10-0-133-32 kubenswrapper[2578]:  >
Jul 31 15:11:12.490460 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:11:12.490228    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:11:12.490460 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:11:12.490460 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:11:12.490460 ip-10-0-133-32 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8" podUID="c63816e9-8042-4842-9d61-8817a972c480" containerName="main" containerID="cri-o://0d40b2d50819e50a37355d579660a7c5de569039cdb1ecf68b02d5758ef54a8a"
Jul 31 15:11:12.490460 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:12.490277    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8" podUID="c63816e9-8042-4842-9d61-8817a972c480" containerName="main" containerID="cri-o://0d40b2d50819e50a37355d579660a7c5de569039cdb1ecf68b02d5758ef54a8a" gracePeriod=60
Jul 31 15:11:12.491696 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:12.491672    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"]
Jul 31 15:11:12.822420 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:12.822311    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8" podUID="c63816e9-8042-4842-9d61-8817a972c480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 31 15:11:12.920597 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:12.920569    2578 generic.go:356] "Generic (PLEG): container finished" podID="c63816e9-8042-4842-9d61-8817a972c480" containerID="0d40b2d50819e50a37355d579660a7c5de569039cdb1ecf68b02d5758ef54a8a" exitCode=0
Jul 31 15:11:12.920752 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:12.920605    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8" event={"ID":"c63816e9-8042-4842-9d61-8817a972c480","Type":"ContainerDied","Data":"0d40b2d50819e50a37355d579660a7c5de569039cdb1ecf68b02d5758ef54a8a"}
Jul 31 15:11:13.246531 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.246505    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:11:13.296895 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.296867    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtsqb\" (UniqueName: \"kubernetes.io/projected/c63816e9-8042-4842-9d61-8817a972c480-kube-api-access-rtsqb\") pod \"c63816e9-8042-4842-9d61-8817a972c480\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") "
Jul 31 15:11:13.297084 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.296916    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-model-cache\") pod \"c63816e9-8042-4842-9d61-8817a972c480\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") "
Jul 31 15:11:13.297084 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.296964    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-dshm\") pod \"c63816e9-8042-4842-9d61-8817a972c480\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") "
Jul 31 15:11:13.297084 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.296987    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-home\") pod \"c63816e9-8042-4842-9d61-8817a972c480\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") "
Jul 31 15:11:13.297084 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.297014    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63816e9-8042-4842-9d61-8817a972c480-tls-certs\") pod \"c63816e9-8042-4842-9d61-8817a972c480\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") "
Jul 31 15:11:13.297084 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.297075    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-tmp-dir\") pod \"c63816e9-8042-4842-9d61-8817a972c480\" (UID: \"c63816e9-8042-4842-9d61-8817a972c480\") "
Jul 31 15:11:13.297384 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.297213    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-home" (OuterVolumeSpecName: "home") pod "c63816e9-8042-4842-9d61-8817a972c480" (UID: "c63816e9-8042-4842-9d61-8817a972c480"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:13.297384 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.297230    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-model-cache" (OuterVolumeSpecName: "model-cache") pod "c63816e9-8042-4842-9d61-8817a972c480" (UID: "c63816e9-8042-4842-9d61-8817a972c480"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:13.297384 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.297343    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:11:13.297538 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.297385    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:11:13.297591 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.297540    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c63816e9-8042-4842-9d61-8817a972c480" (UID: "c63816e9-8042-4842-9d61-8817a972c480"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:13.299207 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.299178    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-dshm" (OuterVolumeSpecName: "dshm") pod "c63816e9-8042-4842-9d61-8817a972c480" (UID: "c63816e9-8042-4842-9d61-8817a972c480"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:13.299505 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.299484    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63816e9-8042-4842-9d61-8817a972c480-kube-api-access-rtsqb" (OuterVolumeSpecName: "kube-api-access-rtsqb") pod "c63816e9-8042-4842-9d61-8817a972c480" (UID: "c63816e9-8042-4842-9d61-8817a972c480"). InnerVolumeSpecName "kube-api-access-rtsqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:11:13.299585 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.299569    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c63816e9-8042-4842-9d61-8817a972c480-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c63816e9-8042-4842-9d61-8817a972c480" (UID: "c63816e9-8042-4842-9d61-8817a972c480"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:11:13.398238 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.398153    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:11:13.398238 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.398185    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63816e9-8042-4842-9d61-8817a972c480-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:11:13.398238 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.398197    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63816e9-8042-4842-9d61-8817a972c480-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:11:13.398238 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.398207    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtsqb\" (UniqueName: \"kubernetes.io/projected/c63816e9-8042-4842-9d61-8817a972c480-kube-api-access-rtsqb\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:11:13.925727 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.925693    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"
Jul 31 15:11:13.925727 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.925706    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8" event={"ID":"c63816e9-8042-4842-9d61-8817a972c480","Type":"ContainerDied","Data":"e645fe00193ae90a396e9af155e8f0c30463d7013953ea824ec725358a64a613"}
Jul 31 15:11:13.926209 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.925772    2578 scope.go:117] "RemoveContainer" containerID="0d40b2d50819e50a37355d579660a7c5de569039cdb1ecf68b02d5758ef54a8a"
Jul 31 15:11:13.951139 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.951109    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"]
Jul 31 15:11:13.957849 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:13.957823    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-vvjj8"]
Jul 31 15:11:14.489655 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:14.489624    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c63816e9-8042-4842-9d61-8817a972c480" path="/var/lib/kubelet/pods/c63816e9-8042-4842-9d61-8817a972c480/volumes"
Jul 31 15:11:15.911704 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:15.911660    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 31 15:11:20.793305 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:20.793208    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w" podUID="d3eba23d-9c53-4c44-91ca-b7f14e584423" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 15:11:25.921403 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:25.921371    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:11:25.929099 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:25.929077    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:11:27.511664 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:27.511621    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w" podUID="d3eba23d-9c53-4c44-91ca-b7f14e584423" containerName="main" containerID="cri-o://b16c265d7ac96e805157aaf04e1c13ba3f211316459aeb5a725d1b3ce58e0b8b" gracePeriod=45
Jul 31 15:11:27.759235 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:27.759211    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"
Jul 31 15:11:27.911992 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:27.911901    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdk7k\" (UniqueName: \"kubernetes.io/projected/d3eba23d-9c53-4c44-91ca-b7f14e584423-kube-api-access-hdk7k\") pod \"d3eba23d-9c53-4c44-91ca-b7f14e584423\" (UID: \"d3eba23d-9c53-4c44-91ca-b7f14e584423\") "
Jul 31 15:11:27.911992 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:27.911950    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3eba23d-9c53-4c44-91ca-b7f14e584423-tls-certs\") pod \"d3eba23d-9c53-4c44-91ca-b7f14e584423\" (UID: \"d3eba23d-9c53-4c44-91ca-b7f14e584423\") "
Jul 31 15:11:27.913949 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:27.913920    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3eba23d-9c53-4c44-91ca-b7f14e584423-kube-api-access-hdk7k" (OuterVolumeSpecName: "kube-api-access-hdk7k") pod "d3eba23d-9c53-4c44-91ca-b7f14e584423" (UID: "d3eba23d-9c53-4c44-91ca-b7f14e584423"). InnerVolumeSpecName "kube-api-access-hdk7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:11:27.914069 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:27.913980    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3eba23d-9c53-4c44-91ca-b7f14e584423-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3eba23d-9c53-4c44-91ca-b7f14e584423" (UID: "d3eba23d-9c53-4c44-91ca-b7f14e584423"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:11:27.983077 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:27.983039    2578 generic.go:356] "Generic (PLEG): container finished" podID="d3eba23d-9c53-4c44-91ca-b7f14e584423" containerID="b16c265d7ac96e805157aaf04e1c13ba3f211316459aeb5a725d1b3ce58e0b8b" exitCode=0
Jul 31 15:11:27.983246 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:27.983106    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"
Jul 31 15:11:27.983246 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:27.983130    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w" event={"ID":"d3eba23d-9c53-4c44-91ca-b7f14e584423","Type":"ContainerDied","Data":"b16c265d7ac96e805157aaf04e1c13ba3f211316459aeb5a725d1b3ce58e0b8b"}
Jul 31 15:11:27.983246 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:27.983179    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w" event={"ID":"d3eba23d-9c53-4c44-91ca-b7f14e584423","Type":"ContainerDied","Data":"f6eb7d729d95672ebac51e99dbbf73aaadd7b5f80dce57e24701f9db8826f05b"}
Jul 31 15:11:27.983246 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:27.983199    2578 scope.go:117] "RemoveContainer" containerID="b16c265d7ac96e805157aaf04e1c13ba3f211316459aeb5a725d1b3ce58e0b8b"
Jul 31 15:11:27.991537 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:27.991519    2578 scope.go:117] "RemoveContainer" containerID="b16c265d7ac96e805157aaf04e1c13ba3f211316459aeb5a725d1b3ce58e0b8b"
Jul 31 15:11:27.991819 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:11:27.991801    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b16c265d7ac96e805157aaf04e1c13ba3f211316459aeb5a725d1b3ce58e0b8b\": container with ID starting with b16c265d7ac96e805157aaf04e1c13ba3f211316459aeb5a725d1b3ce58e0b8b not found: ID does not exist" containerID="b16c265d7ac96e805157aaf04e1c13ba3f211316459aeb5a725d1b3ce58e0b8b"
Jul 31 15:11:27.991882 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:27.991831    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b16c265d7ac96e805157aaf04e1c13ba3f211316459aeb5a725d1b3ce58e0b8b"} err="failed to get container status \"b16c265d7ac96e805157aaf04e1c13ba3f211316459aeb5a725d1b3ce58e0b8b\": rpc error: code = NotFound desc = could not find container \"b16c265d7ac96e805157aaf04e1c13ba3f211316459aeb5a725d1b3ce58e0b8b\": container with ID starting with b16c265d7ac96e805157aaf04e1c13ba3f211316459aeb5a725d1b3ce58e0b8b not found: ID does not exist"
Jul 31 15:11:28.008846 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:28.008811    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"]
Jul 31 15:11:28.012906 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:28.012872    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdk7k\" (UniqueName: \"kubernetes.io/projected/d3eba23d-9c53-4c44-91ca-b7f14e584423-kube-api-access-hdk7k\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:11:28.012906 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:28.012897    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3eba23d-9c53-4c44-91ca-b7f14e584423-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:11:28.013044 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:28.012905    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f98m74w"]
Jul 31 15:11:28.494177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:28.494142    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3eba23d-9c53-4c44-91ca-b7f14e584423" path="/var/lib/kubelet/pods/d3eba23d-9c53-4c44-91ca-b7f14e584423/volumes"
Jul 31 15:11:30.831102 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:30.831063    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"]
Jul 31 15:11:45.853045 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:45.852994    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="main" containerID="cri-o://13188f8d2185303431c94ab61896f16a82cb3d43539e87ce2dbb6f2042491fc4" gracePeriod=45
Jul 31 15:11:45.922139 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:45.922085    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 31 15:11:46.606739 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.606705    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:11:46.772226 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.772186    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-kserve-provision-location\") pod \"5837f3f2-5394-4d2b-92df-f7d9619698c9\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") "
Jul 31 15:11:46.772483 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.772264    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5837f3f2-5394-4d2b-92df-f7d9619698c9-cabundle-cert\") pod \"5837f3f2-5394-4d2b-92df-f7d9619698c9\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") "
Jul 31 15:11:46.772483 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.772291    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-dshm\") pod \"5837f3f2-5394-4d2b-92df-f7d9619698c9\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") "
Jul 31 15:11:46.772483 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.772335    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5837f3f2-5394-4d2b-92df-f7d9619698c9-tls-certs\") pod \"5837f3f2-5394-4d2b-92df-f7d9619698c9\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") "
Jul 31 15:11:46.772483 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.772388    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-model-cache\") pod \"5837f3f2-5394-4d2b-92df-f7d9619698c9\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") "
Jul 31 15:11:46.772483 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.772428    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6fft\" (UniqueName: \"kubernetes.io/projected/5837f3f2-5394-4d2b-92df-f7d9619698c9-kube-api-access-f6fft\") pod \"5837f3f2-5394-4d2b-92df-f7d9619698c9\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") "
Jul 31 15:11:46.772483 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.772465    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-home\") pod \"5837f3f2-5394-4d2b-92df-f7d9619698c9\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") "
Jul 31 15:11:46.772821 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.772495    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-tmp-dir\") pod \"5837f3f2-5394-4d2b-92df-f7d9619698c9\" (UID: \"5837f3f2-5394-4d2b-92df-f7d9619698c9\") "
Jul 31 15:11:46.772821 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.772640    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "5837f3f2-5394-4d2b-92df-f7d9619698c9" (UID: "5837f3f2-5394-4d2b-92df-f7d9619698c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:46.772821 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.772718    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:11:46.773033 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.773006    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5837f3f2-5394-4d2b-92df-f7d9619698c9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5837f3f2-5394-4d2b-92df-f7d9619698c9" (UID: "5837f3f2-5394-4d2b-92df-f7d9619698c9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:11:46.773371 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.773331    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-home" (OuterVolumeSpecName: "home") pod "5837f3f2-5394-4d2b-92df-f7d9619698c9" (UID: "5837f3f2-5394-4d2b-92df-f7d9619698c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:46.774805 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.774764    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5837f3f2-5394-4d2b-92df-f7d9619698c9-kube-api-access-f6fft" (OuterVolumeSpecName: "kube-api-access-f6fft") pod "5837f3f2-5394-4d2b-92df-f7d9619698c9" (UID: "5837f3f2-5394-4d2b-92df-f7d9619698c9"). InnerVolumeSpecName "kube-api-access-f6fft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:11:46.774904 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.774868    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5837f3f2-5394-4d2b-92df-f7d9619698c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5837f3f2-5394-4d2b-92df-f7d9619698c9" (UID: "5837f3f2-5394-4d2b-92df-f7d9619698c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:11:46.774967 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.774951    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-dshm" (OuterVolumeSpecName: "dshm") pod "5837f3f2-5394-4d2b-92df-f7d9619698c9" (UID: "5837f3f2-5394-4d2b-92df-f7d9619698c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:46.787026 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.786997    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5837f3f2-5394-4d2b-92df-f7d9619698c9" (UID: "5837f3f2-5394-4d2b-92df-f7d9619698c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:46.796964 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.796936    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5837f3f2-5394-4d2b-92df-f7d9619698c9" (UID: "5837f3f2-5394-4d2b-92df-f7d9619698c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:46.873871 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.873842    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5837f3f2-5394-4d2b-92df-f7d9619698c9-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:11:46.873871 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.873869    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6fft\" (UniqueName: \"kubernetes.io/projected/5837f3f2-5394-4d2b-92df-f7d9619698c9-kube-api-access-f6fft\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:11:46.873871 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.873877    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:11:46.874271 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.873886    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:11:46.874271 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.873894    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:11:46.874271 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.873902    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5837f3f2-5394-4d2b-92df-f7d9619698c9-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:11:46.874271 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:46.873909    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5837f3f2-5394-4d2b-92df-f7d9619698c9-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:11:47.054412 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:47.054302    2578 generic.go:356] "Generic (PLEG): container finished" podID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerID="13188f8d2185303431c94ab61896f16a82cb3d43539e87ce2dbb6f2042491fc4" exitCode=0
Jul 31 15:11:47.054412 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:47.054360    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"
Jul 31 15:11:47.054596 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:47.054360    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" event={"ID":"5837f3f2-5394-4d2b-92df-f7d9619698c9","Type":"ContainerDied","Data":"13188f8d2185303431c94ab61896f16a82cb3d43539e87ce2dbb6f2042491fc4"}
Jul 31 15:11:47.054596 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:47.054489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87" event={"ID":"5837f3f2-5394-4d2b-92df-f7d9619698c9","Type":"ContainerDied","Data":"20122258c896bc63e2b0691d56accd243a0abda30571fbc9c8974f21166b837a"}
Jul 31 15:11:47.054596 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:47.054519    2578 scope.go:117] "RemoveContainer" containerID="13188f8d2185303431c94ab61896f16a82cb3d43539e87ce2dbb6f2042491fc4"
Jul 31 15:11:47.065003 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:47.064986    2578 scope.go:117] "RemoveContainer" containerID="50779e3949be6a464ac68bd78f8ac72149bda2fc956157d1f46d6257affe1320"
Jul 31 15:11:47.072036 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:47.072019    2578 scope.go:117] "RemoveContainer" containerID="13188f8d2185303431c94ab61896f16a82cb3d43539e87ce2dbb6f2042491fc4"
Jul 31 15:11:47.072278 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:11:47.072261    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13188f8d2185303431c94ab61896f16a82cb3d43539e87ce2dbb6f2042491fc4\": container with ID starting with 13188f8d2185303431c94ab61896f16a82cb3d43539e87ce2dbb6f2042491fc4 not found: ID does not exist" containerID="13188f8d2185303431c94ab61896f16a82cb3d43539e87ce2dbb6f2042491fc4"
Jul 31 15:11:47.072331 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:47.072287    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13188f8d2185303431c94ab61896f16a82cb3d43539e87ce2dbb6f2042491fc4"} err="failed to get container status \"13188f8d2185303431c94ab61896f16a82cb3d43539e87ce2dbb6f2042491fc4\": rpc error: code = NotFound desc = could not find container \"13188f8d2185303431c94ab61896f16a82cb3d43539e87ce2dbb6f2042491fc4\": container with ID starting with 13188f8d2185303431c94ab61896f16a82cb3d43539e87ce2dbb6f2042491fc4 not found: ID does not exist"
Jul 31 15:11:47.072331 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:47.072303    2578 scope.go:117] "RemoveContainer" containerID="50779e3949be6a464ac68bd78f8ac72149bda2fc956157d1f46d6257affe1320"
Jul 31 15:11:47.072634 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:11:47.072562    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50779e3949be6a464ac68bd78f8ac72149bda2fc956157d1f46d6257affe1320\": container with ID starting with 50779e3949be6a464ac68bd78f8ac72149bda2fc956157d1f46d6257affe1320 not found: ID does not exist" containerID="50779e3949be6a464ac68bd78f8ac72149bda2fc956157d1f46d6257affe1320"
Jul 31 15:11:47.072634 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:47.072582    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50779e3949be6a464ac68bd78f8ac72149bda2fc956157d1f46d6257affe1320"} err="failed to get container status \"50779e3949be6a464ac68bd78f8ac72149bda2fc956157d1f46d6257affe1320\": rpc error: code = NotFound desc = could not find container \"50779e3949be6a464ac68bd78f8ac72149bda2fc956157d1f46d6257affe1320\": container with ID starting with 50779e3949be6a464ac68bd78f8ac72149bda2fc956157d1f46d6257affe1320 not found: ID does not exist"
Jul 31 15:11:47.088334 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:47.088312    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"]
Jul 31 15:11:47.096880 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:47.096857    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6996464c95-p4t87"]
Jul 31 15:11:48.464211 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.464177    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"]
Jul 31 15:11:48.464587 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.464465    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3eba23d-9c53-4c44-91ca-b7f14e584423" containerName="main"
Jul 31 15:11:48.464587 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.464477    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3eba23d-9c53-4c44-91ca-b7f14e584423" containerName="main"
Jul 31 15:11:48.464587 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.464490    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="storage-initializer"
Jul 31 15:11:48.464587 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.464495    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="storage-initializer"
Jul 31 15:11:48.464587 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.464501    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="main"
Jul 31 15:11:48.464587 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.464508    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="main"
Jul 31 15:11:48.464587 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.464522    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c63816e9-8042-4842-9d61-8817a972c480" containerName="main"
Jul 31 15:11:48.464587 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.464529    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c63816e9-8042-4842-9d61-8817a972c480" containerName="main"
Jul 31 15:11:48.464587 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.464577    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" containerName="main"
Jul 31 15:11:48.464587 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.464586    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3eba23d-9c53-4c44-91ca-b7f14e584423" containerName="main"
Jul 31 15:11:48.465064 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.464598    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c63816e9-8042-4842-9d61-8817a972c480" containerName="main"
Jul 31 15:11:48.469523 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.469502    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.472405 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.472377    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 31 15:11:48.473620 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.473602    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-n9hbs\""
Jul 31 15:11:48.473738 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.473639    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 31 15:11:48.473738 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.473655    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 31 15:11:48.478705 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.478682    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"]
Jul 31 15:11:48.490028 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.490004    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5837f3f2-5394-4d2b-92df-f7d9619698c9" path="/var/lib/kubelet/pods/5837f3f2-5394-4d2b-92df-f7d9619698c9/volumes"
Jul 31 15:11:48.589649 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.589602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.589815 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.589673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.589815 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.589693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.589815 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.589710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5359bd74-a183-42c4-970c-3c3ebd623288-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.589815 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.589732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.589951 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.589829    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zph9v\" (UniqueName: \"kubernetes.io/projected/5359bd74-a183-42c4-970c-3c3ebd623288-kube-api-access-zph9v\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.690495 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.690335    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zph9v\" (UniqueName: \"kubernetes.io/projected/5359bd74-a183-42c4-970c-3c3ebd623288-kube-api-access-zph9v\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.690709 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.690540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.690709 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.690607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.690709 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.690631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.690709 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.690658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5359bd74-a183-42c4-970c-3c3ebd623288-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.690952 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.690778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.691011 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.690966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.691071 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.691030    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.691128 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.691067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.693001 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.692982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.693278 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.693261    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5359bd74-a183-42c4-970c-3c3ebd623288-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.702765 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.702737    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zph9v\" (UniqueName: \"kubernetes.io/projected/5359bd74-a183-42c4-970c-3c3ebd623288-kube-api-access-zph9v\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.781342 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.781308    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:48.906097 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:48.906064    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"]
Jul 31 15:11:48.908291 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:11:48.908263    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5359bd74_a183_42c4_970c_3c3ebd623288.slice/crio-9b1156d5ff9c656be0093661f48716e7ec4ee96291a3c48a45e7426a59c6db5f WatchSource:0}: Error finding container 9b1156d5ff9c656be0093661f48716e7ec4ee96291a3c48a45e7426a59c6db5f: Status 404 returned error can't find the container with id 9b1156d5ff9c656be0093661f48716e7ec4ee96291a3c48a45e7426a59c6db5f
Jul 31 15:11:49.064643 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:49.064559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq" event={"ID":"5359bd74-a183-42c4-970c-3c3ebd623288","Type":"ContainerStarted","Data":"fe29baf592902927ef3b040685b7496587a34ce865cb37c703995c551c245373"}
Jul 31 15:11:49.064643 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:49.064619    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq" event={"ID":"5359bd74-a183-42c4-970c-3c3ebd623288","Type":"ContainerStarted","Data":"9b1156d5ff9c656be0093661f48716e7ec4ee96291a3c48a45e7426a59c6db5f"}
Jul 31 15:11:49.091974 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:49.091878    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq" podStartSLOduration=1.091853039 podStartE2EDuration="1.091853039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:11:49.084769737 +0000 UTC m=+1407.093981608" watchObservedRunningTime="2026-07-31 15:11:49.091853039 +0000 UTC m=+1407.101064908"
Jul 31 15:11:49.781881 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:49.781845    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:58.781968 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:58.781905    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:58.794405 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:58.794375    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:11:59.112061 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:11:59.111975    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:12:13.314038 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.314003    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"]
Jul 31 15:12:13.317134 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.317115    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.320138 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.320110    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Jul 31 15:12:13.321438 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.321409    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-btsvw\""
Jul 31 15:12:13.321556 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.321444    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Jul 31 15:12:13.321556 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.321418    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Jul 31 15:12:13.328595 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.328570    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"]
Jul 31 15:12:13.367068 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.367033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-dshm\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.367245 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.367075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-home\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.367245 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.367097    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc757396-4b7a-4959-8ba4-c987bb2b6b86-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.367245 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.367114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.367245 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.367177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-model-cache\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.367245 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.367207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhrhh\" (UniqueName: \"kubernetes.io/projected/bc757396-4b7a-4959-8ba4-c987bb2b6b86-kube-api-access-nhrhh\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.468007 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.467956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-home\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.468207 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.468021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc757396-4b7a-4959-8ba4-c987bb2b6b86-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.468207 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.468071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.468207 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.468113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-model-cache\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.468207 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.468136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhrhh\" (UniqueName: \"kubernetes.io/projected/bc757396-4b7a-4959-8ba4-c987bb2b6b86-kube-api-access-nhrhh\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.468470 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.468242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-dshm\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.468711 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.468552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-home\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.468711 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.468615    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-model-cache\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.468711 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.468648    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.470628 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.470601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-dshm\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.470768 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.470749    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc757396-4b7a-4959-8ba4-c987bb2b6b86-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.477636 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.477612    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhrhh\" (UniqueName: \"kubernetes.io/projected/bc757396-4b7a-4959-8ba4-c987bb2b6b86-kube-api-access-nhrhh\") pod \"canary-v1-kserve-65d58ccc9-pdgkq\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.627610 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.627507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:13.762474 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.762440    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"]
Jul 31 15:12:13.764824 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:12:13.764798    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc757396_4b7a_4959_8ba4_c987bb2b6b86.slice/crio-9fe4bc04297539b43a40f9723739ddc0932ae92a97014d99e6f141b00902608a WatchSource:0}: Error finding container 9fe4bc04297539b43a40f9723739ddc0932ae92a97014d99e6f141b00902608a: Status 404 returned error can't find the container with id 9fe4bc04297539b43a40f9723739ddc0932ae92a97014d99e6f141b00902608a
Jul 31 15:12:13.767097 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:13.767079    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:12:14.160110 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:14.160051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq" event={"ID":"bc757396-4b7a-4959-8ba4-c987bb2b6b86","Type":"ContainerStarted","Data":"9fe4bc04297539b43a40f9723739ddc0932ae92a97014d99e6f141b00902608a"}
Jul 31 15:12:15.165783 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:15.165738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq" event={"ID":"bc757396-4b7a-4959-8ba4-c987bb2b6b86","Type":"ContainerStarted","Data":"a8c94fb3cb26fcf1eb3dd4cc2381f02b8c4da0084366eff0198584b08d8c49b9"}
Jul 31 15:12:15.207227 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:15.207126    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq" podStartSLOduration=1.028634848 podStartE2EDuration="2.207108098s" totalImagesPullingTime="1.17847325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:12:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:12:15.206290115 +0000 UTC m=+1433.215501984" watchObservedRunningTime="2026-07-31 15:12:15.207108098 +0000 UTC m=+1433.216319966"
Jul 31 15:12:15.628207 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:15.628163    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:23.628212 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:23.628173    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:23.640421 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:23.640396    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:24.215310 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:24.215282    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:12:33.297593 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.297560    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"]
Jul 31 15:12:33.300575 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.300557    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.303583 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.303555    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Jul 31 15:12:33.311728 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.311702    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"]
Jul 31 15:12:33.330422 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.330391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb9b0de-0a30-45bd-9beb-731f5197f830-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.330580 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.330448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-home\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.330580 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.330469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-dshm\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.330580 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.330521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.330580 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.330552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwwcj\" (UniqueName: \"kubernetes.io/projected/4fb9b0de-0a30-45bd-9beb-731f5197f830-kube-api-access-jwwcj\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.330736 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.330590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.431884 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.431844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.431884 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.431884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwwcj\" (UniqueName: \"kubernetes.io/projected/4fb9b0de-0a30-45bd-9beb-731f5197f830-kube-api-access-jwwcj\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.432152 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.431959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.432152 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.431984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb9b0de-0a30-45bd-9beb-731f5197f830-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.432152 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.432046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-home\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.432152 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.432071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-dshm\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.432421 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.432310    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.432492 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.432419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.432530 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.432485    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-home\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.434285 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.434266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-dshm\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.434562 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.434541    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb9b0de-0a30-45bd-9beb-731f5197f830-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.447266 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.447244    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwwcj\" (UniqueName: \"kubernetes.io/projected/4fb9b0de-0a30-45bd-9beb-731f5197f830-kube-api-access-jwwcj\") pod \"canary-v2-kserve-7f65dbbccc-zgqln\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.611464 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.611375    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:33.746305 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:33.746271    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"]
Jul 31 15:12:33.749814 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:12:33.749781    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb9b0de_0a30_45bd_9beb_731f5197f830.slice/crio-08dd17f1ac586e2a8b4df2a96cc2531f2a4a7acbe1e5b5703d6c15de528bec5a WatchSource:0}: Error finding container 08dd17f1ac586e2a8b4df2a96cc2531f2a4a7acbe1e5b5703d6c15de528bec5a: Status 404 returned error can't find the container with id 08dd17f1ac586e2a8b4df2a96cc2531f2a4a7acbe1e5b5703d6c15de528bec5a
Jul 31 15:12:34.246549 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:34.246430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln" event={"ID":"4fb9b0de-0a30-45bd-9beb-731f5197f830","Type":"ContainerStarted","Data":"fa19a24d7078971f6041e6e9f7c2ace6121eb48309b023eb9e77ff528a7c91b3"}
Jul 31 15:12:34.246549 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:34.246502    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln" event={"ID":"4fb9b0de-0a30-45bd-9beb-731f5197f830","Type":"ContainerStarted","Data":"08dd17f1ac586e2a8b4df2a96cc2531f2a4a7acbe1e5b5703d6c15de528bec5a"}
Jul 31 15:12:34.267592 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:34.267538    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln" podStartSLOduration=1.267517169 podStartE2EDuration="1.267517169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:12:34.266406393 +0000 UTC m=+1452.275618264" watchObservedRunningTime="2026-07-31 15:12:34.267517169 +0000 UTC m=+1452.276729042"
Jul 31 15:12:34.611704 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:34.611619    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:43.612553 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:43.612511    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:43.624889 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:43.624854    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:44.297829 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:44.297803    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:12:56.751850 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:56.751812    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"]
Jul 31 15:12:56.760925 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:12:56.760882    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:12:56.760925 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:12:56.760925 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:12:56.760925 ip-10-0-133-32 kubenswrapper[2578]:  > containerID="fe29baf592902927ef3b040685b7496587a34ce865cb37c703995c551c245373" cmd=["/bin/sleep","15"]
Jul 31 15:12:56.761283 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:56.760941    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq" message="" err=<
Jul 31 15:12:56.761283 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:12:56.761283 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:12:56.761283 ip-10-0-133-32 kubenswrapper[2578]:  >
Jul 31 15:12:56.761283 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:12:56.760954    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:12:56.761283 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:12:56.761283 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:12:56.761283 ip-10-0-133-32 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq" podUID="5359bd74-a183-42c4-970c-3c3ebd623288" containerName="main" containerID="cri-o://fe29baf592902927ef3b040685b7496587a34ce865cb37c703995c551c245373"
Jul 31 15:12:56.761283 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:56.760985    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq" podUID="5359bd74-a183-42c4-970c-3c3ebd623288" containerName="main" containerID="cri-o://fe29baf592902927ef3b040685b7496587a34ce865cb37c703995c551c245373" gracePeriod=60
Jul 31 15:12:57.109466 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.109438    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:12:57.235505 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.235457    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5359bd74-a183-42c4-970c-3c3ebd623288-tls-certs\") pod \"5359bd74-a183-42c4-970c-3c3ebd623288\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") "
Jul 31 15:12:57.235699 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.235532    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-dshm\") pod \"5359bd74-a183-42c4-970c-3c3ebd623288\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") "
Jul 31 15:12:57.235699 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.235620    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-model-cache\") pod \"5359bd74-a183-42c4-970c-3c3ebd623288\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") "
Jul 31 15:12:57.235699 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.235689    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-home\") pod \"5359bd74-a183-42c4-970c-3c3ebd623288\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") "
Jul 31 15:12:57.235872 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.235714    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-tmp-dir\") pod \"5359bd74-a183-42c4-970c-3c3ebd623288\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") "
Jul 31 15:12:57.235872 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.235743    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zph9v\" (UniqueName: \"kubernetes.io/projected/5359bd74-a183-42c4-970c-3c3ebd623288-kube-api-access-zph9v\") pod \"5359bd74-a183-42c4-970c-3c3ebd623288\" (UID: \"5359bd74-a183-42c4-970c-3c3ebd623288\") "
Jul 31 15:12:57.235987 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.235875    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-model-cache" (OuterVolumeSpecName: "model-cache") pod "5359bd74-a183-42c4-970c-3c3ebd623288" (UID: "5359bd74-a183-42c4-970c-3c3ebd623288"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:12:57.236039 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.235982    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-home" (OuterVolumeSpecName: "home") pod "5359bd74-a183-42c4-970c-3c3ebd623288" (UID: "5359bd74-a183-42c4-970c-3c3ebd623288"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:12:57.236039 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.235992    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5359bd74-a183-42c4-970c-3c3ebd623288" (UID: "5359bd74-a183-42c4-970c-3c3ebd623288"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:12:57.236152 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.236047    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:12:57.237791 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.237765    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-dshm" (OuterVolumeSpecName: "dshm") pod "5359bd74-a183-42c4-970c-3c3ebd623288" (UID: "5359bd74-a183-42c4-970c-3c3ebd623288"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:12:57.237927 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.237812    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5359bd74-a183-42c4-970c-3c3ebd623288-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5359bd74-a183-42c4-970c-3c3ebd623288" (UID: "5359bd74-a183-42c4-970c-3c3ebd623288"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:12:57.237927 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.237902    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5359bd74-a183-42c4-970c-3c3ebd623288-kube-api-access-zph9v" (OuterVolumeSpecName: "kube-api-access-zph9v") pod "5359bd74-a183-42c4-970c-3c3ebd623288" (UID: "5359bd74-a183-42c4-970c-3c3ebd623288"). InnerVolumeSpecName "kube-api-access-zph9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:12:57.337450 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.337370    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:12:57.337450 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.337402    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:12:57.337450 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.337411    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5359bd74-a183-42c4-970c-3c3ebd623288-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:12:57.337450 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.337421    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zph9v\" (UniqueName: \"kubernetes.io/projected/5359bd74-a183-42c4-970c-3c3ebd623288-kube-api-access-zph9v\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:12:57.337450 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.337430    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5359bd74-a183-42c4-970c-3c3ebd623288-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:12:57.347168 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.347138    2578 generic.go:356] "Generic (PLEG): container finished" podID="5359bd74-a183-42c4-970c-3c3ebd623288" containerID="fe29baf592902927ef3b040685b7496587a34ce865cb37c703995c551c245373" exitCode=0
Jul 31 15:12:57.347313 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.347171    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq" event={"ID":"5359bd74-a183-42c4-970c-3c3ebd623288","Type":"ContainerDied","Data":"fe29baf592902927ef3b040685b7496587a34ce865cb37c703995c551c245373"}
Jul 31 15:12:57.347313 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.347189    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"
Jul 31 15:12:57.347313 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.347252    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq" event={"ID":"5359bd74-a183-42c4-970c-3c3ebd623288","Type":"ContainerDied","Data":"9b1156d5ff9c656be0093661f48716e7ec4ee96291a3c48a45e7426a59c6db5f"}
Jul 31 15:12:57.347313 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.347274    2578 scope.go:117] "RemoveContainer" containerID="fe29baf592902927ef3b040685b7496587a34ce865cb37c703995c551c245373"
Jul 31 15:12:57.355593 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.355578    2578 scope.go:117] "RemoveContainer" containerID="fe29baf592902927ef3b040685b7496587a34ce865cb37c703995c551c245373"
Jul 31 15:12:57.355830 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:12:57.355806    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe29baf592902927ef3b040685b7496587a34ce865cb37c703995c551c245373\": container with ID starting with fe29baf592902927ef3b040685b7496587a34ce865cb37c703995c551c245373 not found: ID does not exist" containerID="fe29baf592902927ef3b040685b7496587a34ce865cb37c703995c551c245373"
Jul 31 15:12:57.355919 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.355833    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe29baf592902927ef3b040685b7496587a34ce865cb37c703995c551c245373"} err="failed to get container status \"fe29baf592902927ef3b040685b7496587a34ce865cb37c703995c551c245373\": rpc error: code = NotFound desc = could not find container \"fe29baf592902927ef3b040685b7496587a34ce865cb37c703995c551c245373\": container with ID starting with fe29baf592902927ef3b040685b7496587a34ce865cb37c703995c551c245373 not found: ID does not exist"
Jul 31 15:12:57.381650 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.381626    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"]
Jul 31 15:12:57.395281 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:57.395256    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq"]
Jul 31 15:12:58.102453 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:58.102405    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fkztzq" podUID="5359bd74-a183-42c4-970c-3c3ebd623288" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 31 15:12:58.489502 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:12:58.489470    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5359bd74-a183-42c4-970c-3c3ebd623288" path="/var/lib/kubelet/pods/5359bd74-a183-42c4-970c-3c3ebd623288/volumes"
Jul 31 15:13:23.994166 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:23.994133    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"]
Jul 31 15:13:23.994599 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:23.994426    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5359bd74-a183-42c4-970c-3c3ebd623288" containerName="main"
Jul 31 15:13:23.994599 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:23.994437    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5359bd74-a183-42c4-970c-3c3ebd623288" containerName="main"
Jul 31 15:13:23.994599 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:23.994493    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5359bd74-a183-42c4-970c-3c3ebd623288" containerName="main"
Jul 31 15:13:23.997173 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:23.997156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.000743 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.000718    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:13:24.000876 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.000797    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 31 15:13:24.000876 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.000833    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 31 15:13:24.000876 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.000860    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 31 15:13:24.001987 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.001972    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 31 15:13:24.002086 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.002072    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-h8q2s\""
Jul 31 15:13:24.010961 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.010938    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"]
Jul 31 15:13:24.045662 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.045637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.045795 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.045668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.045795 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.045693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp865\" (UniqueName: \"kubernetes.io/projected/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-kube-api-access-jp865\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.045795 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.045710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-dshm\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.045795 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.045767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.045931 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.045824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-model-cache\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.045931 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.045869    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.045931 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.045896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-home\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.146272 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.146229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.146272 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.146279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-home\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.146543 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.146306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.146543 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.146325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.146543 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.146376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp865\" (UniqueName: \"kubernetes.io/projected/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-kube-api-access-jp865\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.146543 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.146404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-dshm\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.146543 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.146430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.146543 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.146477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-model-cache\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.146858 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.146772    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-home\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.146858 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.146812    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.147040 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.147011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.147179 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.147045    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-model-cache\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.147311 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.147293    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.148588 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.148568    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-dshm\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.148716 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.148698    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.155175 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.155151    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp865\" (UniqueName: \"kubernetes.io/projected/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-kube-api-access-jp865\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.254329 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.254250    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"]
Jul 31 15:13:24.258312 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.258295    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:13:24.268969 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.268941    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"]
Jul 31 15:13:24.307587 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.307550    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:24.348605 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.348575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4376a9b4-95bd-4fbd-998e-0649d0d6a323-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd\" (UID: \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:13:24.348744 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.348611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4376a9b4-95bd-4fbd-998e-0649d0d6a323-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd\" (UID: \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:13:24.348744 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.348706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4376a9b4-95bd-4fbd-998e-0649d0d6a323-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd\" (UID: \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:13:24.348744 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.348730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4376a9b4-95bd-4fbd-998e-0649d0d6a323-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd\" (UID: \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:13:24.438545 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.438507    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"]
Jul 31 15:13:24.442716 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:13:24.442675    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2e6c85_a46e_4b3b_b4da_da97f0177a30.slice/crio-cca071af5dcf76f6f7940e3292d9ff07f2d7de53953034acf9db07f0f3d65184 WatchSource:0}: Error finding container cca071af5dcf76f6f7940e3292d9ff07f2d7de53953034acf9db07f0f3d65184: Status 404 returned error can't find the container with id cca071af5dcf76f6f7940e3292d9ff07f2d7de53953034acf9db07f0f3d65184
Jul 31 15:13:24.450332 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.449672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4376a9b4-95bd-4fbd-998e-0649d0d6a323-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd\" (UID: \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:13:24.450332 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.449728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4376a9b4-95bd-4fbd-998e-0649d0d6a323-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd\" (UID: \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:13:24.450332 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.449787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4376a9b4-95bd-4fbd-998e-0649d0d6a323-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd\" (UID: \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:13:24.450332 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.449822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4376a9b4-95bd-4fbd-998e-0649d0d6a323-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd\" (UID: \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:13:24.453489 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.451155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4376a9b4-95bd-4fbd-998e-0649d0d6a323-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd\" (UID: \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:13:24.453489 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.451411    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4376a9b4-95bd-4fbd-998e-0649d0d6a323-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd\" (UID: \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:13:24.453489 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.451985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4376a9b4-95bd-4fbd-998e-0649d0d6a323-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd\" (UID: \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:13:24.455284 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.455228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp" event={"ID":"1e2e6c85-a46e-4b3b-b4da-da97f0177a30","Type":"ContainerStarted","Data":"cca071af5dcf76f6f7940e3292d9ff07f2d7de53953034acf9db07f0f3d65184"}
Jul 31 15:13:24.457063 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.457043    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4376a9b4-95bd-4fbd-998e-0649d0d6a323-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd\" (UID: \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:13:24.569374 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.569281    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:13:24.703451 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:24.703412    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"]
Jul 31 15:13:24.707753 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:13:24.707724    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4376a9b4_95bd_4fbd_998e_0649d0d6a323.slice/crio-6d44e3df0d58decf8c197b8958eadabf9c87ab492ed57b12a375aa1018e9a40f WatchSource:0}: Error finding container 6d44e3df0d58decf8c197b8958eadabf9c87ab492ed57b12a375aa1018e9a40f: Status 404 returned error can't find the container with id 6d44e3df0d58decf8c197b8958eadabf9c87ab492ed57b12a375aa1018e9a40f
Jul 31 15:13:25.463005 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:25.462961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp" event={"ID":"1e2e6c85-a46e-4b3b-b4da-da97f0177a30","Type":"ContainerStarted","Data":"d07e3baf04378ada1ec563c2b56660e0f32e2b040ad80bf98e9455097b08f870"}
Jul 31 15:13:25.464774 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:25.464742    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd" event={"ID":"4376a9b4-95bd-4fbd-998e-0649d0d6a323","Type":"ContainerStarted","Data":"0e948b0ec3c3c264180366c8db938597fbdb20cb32ccefc35e1804f5c6a42654"}
Jul 31 15:13:25.464884 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:25.464798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd" event={"ID":"4376a9b4-95bd-4fbd-998e-0649d0d6a323","Type":"ContainerStarted","Data":"6d44e3df0d58decf8c197b8958eadabf9c87ab492ed57b12a375aa1018e9a40f"}
Jul 31 15:13:27.474712 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:27.474677    2578 generic.go:356] "Generic (PLEG): container finished" podID="4376a9b4-95bd-4fbd-998e-0649d0d6a323" containerID="0e948b0ec3c3c264180366c8db938597fbdb20cb32ccefc35e1804f5c6a42654" exitCode=0
Jul 31 15:13:27.475147 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:27.474753    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd" event={"ID":"4376a9b4-95bd-4fbd-998e-0649d0d6a323","Type":"ContainerDied","Data":"0e948b0ec3c3c264180366c8db938597fbdb20cb32ccefc35e1804f5c6a42654"}
Jul 31 15:13:27.476321 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:27.476265    2578 generic.go:356] "Generic (PLEG): container finished" podID="1e2e6c85-a46e-4b3b-b4da-da97f0177a30" containerID="d07e3baf04378ada1ec563c2b56660e0f32e2b040ad80bf98e9455097b08f870" exitCode=0
Jul 31 15:13:27.476321 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:27.476295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp" event={"ID":"1e2e6c85-a46e-4b3b-b4da-da97f0177a30","Type":"ContainerDied","Data":"d07e3baf04378ada1ec563c2b56660e0f32e2b040ad80bf98e9455097b08f870"}
Jul 31 15:13:28.491229 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:28.491193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp" event={"ID":"1e2e6c85-a46e-4b3b-b4da-da97f0177a30","Type":"ContainerStarted","Data":"d9863915e74707ae3d6a6bad8e049dc9b98e18ff61847295c32d8973b5ab7e58"}
Jul 31 15:13:28.513322 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:28.513265    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp" podStartSLOduration=2.998272392 podStartE2EDuration="5.513243148s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.514970756s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:13:28.51212284 +0000 UTC m=+1506.521334709" watchObservedRunningTime="2026-07-31 15:13:28.513243148 +0000 UTC m=+1506.522455020"
Jul 31 15:13:29.308476 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:29.308428    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:34.308422 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:34.308384    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:34.321212 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:34.321183    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:13:34.528629 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:13:34.528597    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:14:05.655367 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:05.655325    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd" event={"ID":"4376a9b4-95bd-4fbd-998e-0649d0d6a323","Type":"ContainerStarted","Data":"72976a88503f015ed34724a828359446bcd6d2f6322067778bfa24af5f4937c1"}
Jul 31 15:14:05.655813 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:05.655542    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:14:05.677792 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:05.677748    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd" podStartSLOduration=1.936380651 podStartE2EDuration="41.677734169s" totalImagesPullingTime="37.227136656s" totalInitContainerRuntime="2.514216862s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:13:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:14:05.676047651 +0000 UTC m=+1543.685259523" watchObservedRunningTime="2026-07-31 15:14:05.677734169 +0000 UTC m=+1543.686946064"
Jul 31 15:14:25.284108 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:25.284067    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"]
Jul 31 15:14:25.315695 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:25.288586    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"]
Jul 31 15:14:36.671501 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:36.671466    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:14:40.303680 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.303633    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq" podUID="bc757396-4b7a-4959-8ba4-c987bb2b6b86" containerName="main" containerID="cri-o://a8c94fb3cb26fcf1eb3dd4cc2381f02b8c4da0084366eff0198584b08d8c49b9" gracePeriod=45
Jul 31 15:14:40.335387 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.335327    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln" podUID="4fb9b0de-0a30-45bd-9beb-731f5197f830" containerName="main" containerID="cri-o://fa19a24d7078971f6041e6e9f7c2ace6121eb48309b023eb9e77ff528a7c91b3" gracePeriod=45
Jul 31 15:14:40.588052 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.588020    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:14:40.616678 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.616648    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:14:40.742242 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.742206    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-model-cache\") pod \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") "
Jul 31 15:14:40.742462 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.742263    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-dshm\") pod \"4fb9b0de-0a30-45bd-9beb-731f5197f830\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") "
Jul 31 15:14:40.742462 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.742293    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwwcj\" (UniqueName: \"kubernetes.io/projected/4fb9b0de-0a30-45bd-9beb-731f5197f830-kube-api-access-jwwcj\") pod \"4fb9b0de-0a30-45bd-9beb-731f5197f830\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") "
Jul 31 15:14:40.742462 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.742325    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb9b0de-0a30-45bd-9beb-731f5197f830-tls-certs\") pod \"4fb9b0de-0a30-45bd-9beb-731f5197f830\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") "
Jul 31 15:14:40.742462 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.742368    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-dshm\") pod \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") "
Jul 31 15:14:40.742462 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.742396    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-home\") pod \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") "
Jul 31 15:14:40.742462 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.742432    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-model-cache\") pod \"4fb9b0de-0a30-45bd-9beb-731f5197f830\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") "
Jul 31 15:14:40.742789 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.742518    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc757396-4b7a-4959-8ba4-c987bb2b6b86" (UID: "bc757396-4b7a-4959-8ba4-c987bb2b6b86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:40.742789 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.742687    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-model-cache" (OuterVolumeSpecName: "model-cache") pod "4fb9b0de-0a30-45bd-9beb-731f5197f830" (UID: "4fb9b0de-0a30-45bd-9beb-731f5197f830"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:40.743908 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.743203    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhrhh\" (UniqueName: \"kubernetes.io/projected/bc757396-4b7a-4959-8ba4-c987bb2b6b86-kube-api-access-nhrhh\") pod \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") "
Jul 31 15:14:40.743908 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.743266    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc757396-4b7a-4959-8ba4-c987bb2b6b86-tls-certs\") pod \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") "
Jul 31 15:14:40.743908 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.743295    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-tmp-dir\") pod \"4fb9b0de-0a30-45bd-9beb-731f5197f830\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") "
Jul 31 15:14:40.743908 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.743331    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-tmp-dir\") pod \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\" (UID: \"bc757396-4b7a-4959-8ba4-c987bb2b6b86\") "
Jul 31 15:14:40.743908 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.743366    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-home\") pod \"4fb9b0de-0a30-45bd-9beb-731f5197f830\" (UID: \"4fb9b0de-0a30-45bd-9beb-731f5197f830\") "
Jul 31 15:14:40.743908 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.743629    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:40.743908 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.743323    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-home" (OuterVolumeSpecName: "home") pod "bc757396-4b7a-4959-8ba4-c987bb2b6b86" (UID: "bc757396-4b7a-4959-8ba4-c987bb2b6b86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:40.743908 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.743647    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:40.743908 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.743496    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4fb9b0de-0a30-45bd-9beb-731f5197f830" (UID: "4fb9b0de-0a30-45bd-9beb-731f5197f830"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:40.743908 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.743630    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc757396-4b7a-4959-8ba4-c987bb2b6b86" (UID: "bc757396-4b7a-4959-8ba4-c987bb2b6b86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:40.743908 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.743866    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-home" (OuterVolumeSpecName: "home") pod "4fb9b0de-0a30-45bd-9beb-731f5197f830" (UID: "4fb9b0de-0a30-45bd-9beb-731f5197f830"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:40.745754 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.745723    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fb9b0de-0a30-45bd-9beb-731f5197f830-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fb9b0de-0a30-45bd-9beb-731f5197f830" (UID: "4fb9b0de-0a30-45bd-9beb-731f5197f830"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:14:40.745876 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.745805    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-dshm" (OuterVolumeSpecName: "dshm") pod "4fb9b0de-0a30-45bd-9beb-731f5197f830" (UID: "4fb9b0de-0a30-45bd-9beb-731f5197f830"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:40.746295 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.746253    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-dshm" (OuterVolumeSpecName: "dshm") pod "bc757396-4b7a-4959-8ba4-c987bb2b6b86" (UID: "bc757396-4b7a-4959-8ba4-c987bb2b6b86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:40.746645 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.746604    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb9b0de-0a30-45bd-9beb-731f5197f830-kube-api-access-jwwcj" (OuterVolumeSpecName: "kube-api-access-jwwcj") pod "4fb9b0de-0a30-45bd-9beb-731f5197f830" (UID: "4fb9b0de-0a30-45bd-9beb-731f5197f830"). InnerVolumeSpecName "kube-api-access-jwwcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:14:40.747854 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.747828    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc757396-4b7a-4959-8ba4-c987bb2b6b86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc757396-4b7a-4959-8ba4-c987bb2b6b86" (UID: "bc757396-4b7a-4959-8ba4-c987bb2b6b86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:14:40.748546 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.748518    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc757396-4b7a-4959-8ba4-c987bb2b6b86-kube-api-access-nhrhh" (OuterVolumeSpecName: "kube-api-access-nhrhh") pod "bc757396-4b7a-4959-8ba4-c987bb2b6b86" (UID: "bc757396-4b7a-4959-8ba4-c987bb2b6b86"). InnerVolumeSpecName "kube-api-access-nhrhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:14:40.805228 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.805192    2578 generic.go:356] "Generic (PLEG): container finished" podID="4fb9b0de-0a30-45bd-9beb-731f5197f830" containerID="fa19a24d7078971f6041e6e9f7c2ace6121eb48309b023eb9e77ff528a7c91b3" exitCode=0
Jul 31 15:14:40.805429 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.805238    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"
Jul 31 15:14:40.805429 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.805278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln" event={"ID":"4fb9b0de-0a30-45bd-9beb-731f5197f830","Type":"ContainerDied","Data":"fa19a24d7078971f6041e6e9f7c2ace6121eb48309b023eb9e77ff528a7c91b3"}
Jul 31 15:14:40.805429 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.805327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln" event={"ID":"4fb9b0de-0a30-45bd-9beb-731f5197f830","Type":"ContainerDied","Data":"08dd17f1ac586e2a8b4df2a96cc2531f2a4a7acbe1e5b5703d6c15de528bec5a"}
Jul 31 15:14:40.805429 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.805376    2578 scope.go:117] "RemoveContainer" containerID="fa19a24d7078971f6041e6e9f7c2ace6121eb48309b023eb9e77ff528a7c91b3"
Jul 31 15:14:40.806658 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.806630    2578 generic.go:356] "Generic (PLEG): container finished" podID="bc757396-4b7a-4959-8ba4-c987bb2b6b86" containerID="a8c94fb3cb26fcf1eb3dd4cc2381f02b8c4da0084366eff0198584b08d8c49b9" exitCode=0
Jul 31 15:14:40.806771 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.806661    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq" event={"ID":"bc757396-4b7a-4959-8ba4-c987bb2b6b86","Type":"ContainerDied","Data":"a8c94fb3cb26fcf1eb3dd4cc2381f02b8c4da0084366eff0198584b08d8c49b9"}
Jul 31 15:14:40.806771 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.806670    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"
Jul 31 15:14:40.806771 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.806716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq" event={"ID":"bc757396-4b7a-4959-8ba4-c987bb2b6b86","Type":"ContainerDied","Data":"9fe4bc04297539b43a40f9723739ddc0932ae92a97014d99e6f141b00902608a"}
Jul 31 15:14:40.814067 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.814045    2578 scope.go:117] "RemoveContainer" containerID="fa19a24d7078971f6041e6e9f7c2ace6121eb48309b023eb9e77ff528a7c91b3"
Jul 31 15:14:40.814331 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:14:40.814308    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa19a24d7078971f6041e6e9f7c2ace6121eb48309b023eb9e77ff528a7c91b3\": container with ID starting with fa19a24d7078971f6041e6e9f7c2ace6121eb48309b023eb9e77ff528a7c91b3 not found: ID does not exist" containerID="fa19a24d7078971f6041e6e9f7c2ace6121eb48309b023eb9e77ff528a7c91b3"
Jul 31 15:14:40.814433 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.814343    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa19a24d7078971f6041e6e9f7c2ace6121eb48309b023eb9e77ff528a7c91b3"} err="failed to get container status \"fa19a24d7078971f6041e6e9f7c2ace6121eb48309b023eb9e77ff528a7c91b3\": rpc error: code = NotFound desc = could not find container \"fa19a24d7078971f6041e6e9f7c2ace6121eb48309b023eb9e77ff528a7c91b3\": container with ID starting with fa19a24d7078971f6041e6e9f7c2ace6121eb48309b023eb9e77ff528a7c91b3 not found: ID does not exist"
Jul 31 15:14:40.814433 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.814383    2578 scope.go:117] "RemoveContainer" containerID="a8c94fb3cb26fcf1eb3dd4cc2381f02b8c4da0084366eff0198584b08d8c49b9"
Jul 31 15:14:40.822039 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.822020    2578 scope.go:117] "RemoveContainer" containerID="a8c94fb3cb26fcf1eb3dd4cc2381f02b8c4da0084366eff0198584b08d8c49b9"
Jul 31 15:14:40.822265 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:14:40.822246    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c94fb3cb26fcf1eb3dd4cc2381f02b8c4da0084366eff0198584b08d8c49b9\": container with ID starting with a8c94fb3cb26fcf1eb3dd4cc2381f02b8c4da0084366eff0198584b08d8c49b9 not found: ID does not exist" containerID="a8c94fb3cb26fcf1eb3dd4cc2381f02b8c4da0084366eff0198584b08d8c49b9"
Jul 31 15:14:40.822309 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.822270    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c94fb3cb26fcf1eb3dd4cc2381f02b8c4da0084366eff0198584b08d8c49b9"} err="failed to get container status \"a8c94fb3cb26fcf1eb3dd4cc2381f02b8c4da0084366eff0198584b08d8c49b9\": rpc error: code = NotFound desc = could not find container \"a8c94fb3cb26fcf1eb3dd4cc2381f02b8c4da0084366eff0198584b08d8c49b9\": container with ID starting with a8c94fb3cb26fcf1eb3dd4cc2381f02b8c4da0084366eff0198584b08d8c49b9 not found: ID does not exist"
Jul 31 15:14:40.829566 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.829546    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"]
Jul 31 15:14:40.834424 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.834405    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-zgqln"]
Jul 31 15:14:40.844408 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.844386    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhrhh\" (UniqueName: \"kubernetes.io/projected/bc757396-4b7a-4959-8ba4-c987bb2b6b86-kube-api-access-nhrhh\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:40.844480 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.844409    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc757396-4b7a-4959-8ba4-c987bb2b6b86-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:40.844480 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.844420    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:40.844480 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.844428    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:40.844480 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.844436    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:40.844480 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.844443    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fb9b0de-0a30-45bd-9beb-731f5197f830-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:40.844480 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.844451    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwwcj\" (UniqueName: \"kubernetes.io/projected/4fb9b0de-0a30-45bd-9beb-731f5197f830-kube-api-access-jwwcj\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:40.844480 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.844459    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb9b0de-0a30-45bd-9beb-731f5197f830-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:40.844480 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.844467    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:40.844480 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.844473    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc757396-4b7a-4959-8ba4-c987bb2b6b86-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:40.845496 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.845479    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"]
Jul 31 15:14:40.849778 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:40.849760    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pdgkq"]
Jul 31 15:14:42.489876 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:42.489843    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fb9b0de-0a30-45bd-9beb-731f5197f830" path="/var/lib/kubelet/pods/4fb9b0de-0a30-45bd-9beb-731f5197f830/volumes"
Jul 31 15:14:42.490264 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:42.490248    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc757396-4b7a-4959-8ba4-c987bb2b6b86" path="/var/lib/kubelet/pods/bc757396-4b7a-4959-8ba4-c987bb2b6b86/volumes"
Jul 31 15:14:55.667699 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:55.667669    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd_4376a9b4-95bd-4fbd-998e-0649d0d6a323/main/0.log"
Jul 31 15:14:56.733021 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:56.732978    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"]
Jul 31 15:14:56.733496 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:56.733210    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd" podUID="4376a9b4-95bd-4fbd-998e-0649d0d6a323" containerName="main" containerID="cri-o://72976a88503f015ed34724a828359446bcd6d2f6322067778bfa24af5f4937c1" gracePeriod=30
Jul 31 15:14:56.759870 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:56.759832    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"]
Jul 31 15:14:56.879078 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:14:56.879036    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:14:56.879078 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:14:56.879078 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:14:56.879078 ip-10-0-133-32 kubenswrapper[2578]:  > containerID="d9863915e74707ae3d6a6bad8e049dc9b98e18ff61847295c32d8973b5ab7e58" cmd=["/bin/sleep","15"]
Jul 31 15:14:56.879405 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:56.879084    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp" message="" err=<
Jul 31 15:14:56.879405 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:14:56.879405 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:14:56.879405 ip-10-0-133-32 kubenswrapper[2578]:  >
Jul 31 15:14:56.879405 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:14:56.879096    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:14:56.879405 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:14:56.879405 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:14:56.879405 ip-10-0-133-32 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp" podUID="1e2e6c85-a46e-4b3b-b4da-da97f0177a30" containerName="main" containerID="cri-o://d9863915e74707ae3d6a6bad8e049dc9b98e18ff61847295c32d8973b5ab7e58"
Jul 31 15:14:56.879405 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:56.879136    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp" podUID="1e2e6c85-a46e-4b3b-b4da-da97f0177a30" containerName="main" containerID="cri-o://d9863915e74707ae3d6a6bad8e049dc9b98e18ff61847295c32d8973b5ab7e58" gracePeriod=60
Jul 31 15:14:57.254632 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.254608    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:14:57.284902 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.284873    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-cabundle-cert\") pod \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") "
Jul 31 15:14:57.285076 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.284917    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp865\" (UniqueName: \"kubernetes.io/projected/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-kube-api-access-jp865\") pod \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") "
Jul 31 15:14:57.285149 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.285077    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-dshm\") pod \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") "
Jul 31 15:14:57.285149 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.285120    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-home\") pod \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") "
Jul 31 15:14:57.285260 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.285153    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-tls-certs\") pod \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") "
Jul 31 15:14:57.285260 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.285207    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-tmp-dir\") pod \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") "
Jul 31 15:14:57.285260 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.285242    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-model-cache\") pod \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") "
Jul 31 15:14:57.285260 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.285242    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1e2e6c85-a46e-4b3b-b4da-da97f0177a30" (UID: "1e2e6c85-a46e-4b3b-b4da-da97f0177a30"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:14:57.285517 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.285269    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-kserve-provision-location\") pod \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\" (UID: \"1e2e6c85-a46e-4b3b-b4da-da97f0177a30\") "
Jul 31 15:14:57.285517 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.285425    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e2e6c85-a46e-4b3b-b4da-da97f0177a30" (UID: "1e2e6c85-a46e-4b3b-b4da-da97f0177a30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:57.285623 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.285508    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-home" (OuterVolumeSpecName: "home") pod "1e2e6c85-a46e-4b3b-b4da-da97f0177a30" (UID: "1e2e6c85-a46e-4b3b-b4da-da97f0177a30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:57.285623 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.285536    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.285623 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.285553    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.285768 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.285625    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e2e6c85-a46e-4b3b-b4da-da97f0177a30" (UID: "1e2e6c85-a46e-4b3b-b4da-da97f0177a30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:57.287607 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.287573    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-dshm" (OuterVolumeSpecName: "dshm") pod "1e2e6c85-a46e-4b3b-b4da-da97f0177a30" (UID: "1e2e6c85-a46e-4b3b-b4da-da97f0177a30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:57.287723 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.287625    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-kube-api-access-jp865" (OuterVolumeSpecName: "kube-api-access-jp865") pod "1e2e6c85-a46e-4b3b-b4da-da97f0177a30" (UID: "1e2e6c85-a46e-4b3b-b4da-da97f0177a30"). InnerVolumeSpecName "kube-api-access-jp865". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:14:57.287889 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.287871    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e2e6c85-a46e-4b3b-b4da-da97f0177a30" (UID: "1e2e6c85-a46e-4b3b-b4da-da97f0177a30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:14:57.309044 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.309013    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1e2e6c85-a46e-4b3b-b4da-da97f0177a30" (UID: "1e2e6c85-a46e-4b3b-b4da-da97f0177a30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:57.384903 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.384871    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:14:57.386384 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.386365    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4376a9b4-95bd-4fbd-998e-0649d0d6a323-tls-certs\") pod \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\" (UID: \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\") "
Jul 31 15:14:57.386474 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.386397    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4376a9b4-95bd-4fbd-998e-0649d0d6a323-cabundle-cert\") pod \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\" (UID: \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\") "
Jul 31 15:14:57.386474 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.386448    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4376a9b4-95bd-4fbd-998e-0649d0d6a323-kserve-provision-location\") pod \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\" (UID: \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\") "
Jul 31 15:14:57.386474 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.386469    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4376a9b4-95bd-4fbd-998e-0649d0d6a323-tmp-dir\") pod \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\" (UID: \"4376a9b4-95bd-4fbd-998e-0649d0d6a323\") "
Jul 31 15:14:57.386641 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.386622    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp865\" (UniqueName: \"kubernetes.io/projected/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-kube-api-access-jp865\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.386641 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.386633    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.386641 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.386642    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.386784 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.386650    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.386784 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.386659    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.386784 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.386667    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e2e6c85-a46e-4b3b-b4da-da97f0177a30-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.386891 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.386802    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4376a9b4-95bd-4fbd-998e-0649d0d6a323-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4376a9b4-95bd-4fbd-998e-0649d0d6a323" (UID: "4376a9b4-95bd-4fbd-998e-0649d0d6a323"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:14:57.386891 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.386818    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4376a9b4-95bd-4fbd-998e-0649d0d6a323-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4376a9b4-95bd-4fbd-998e-0649d0d6a323" (UID: "4376a9b4-95bd-4fbd-998e-0649d0d6a323"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:57.388380 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.388360    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4376a9b4-95bd-4fbd-998e-0649d0d6a323-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4376a9b4-95bd-4fbd-998e-0649d0d6a323" (UID: "4376a9b4-95bd-4fbd-998e-0649d0d6a323"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:14:57.408189 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.408147    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4376a9b4-95bd-4fbd-998e-0649d0d6a323-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4376a9b4-95bd-4fbd-998e-0649d0d6a323" (UID: "4376a9b4-95bd-4fbd-998e-0649d0d6a323"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:57.487856 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.487770    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4376a9b4-95bd-4fbd-998e-0649d0d6a323-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.487856 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.487802    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4376a9b4-95bd-4fbd-998e-0649d0d6a323-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.487856 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.487812    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4376a9b4-95bd-4fbd-998e-0649d0d6a323-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.487856 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.487821    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4376a9b4-95bd-4fbd-998e-0649d0d6a323-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.876542 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.876448    2578 generic.go:356] "Generic (PLEG): container finished" podID="1e2e6c85-a46e-4b3b-b4da-da97f0177a30" containerID="d9863915e74707ae3d6a6bad8e049dc9b98e18ff61847295c32d8973b5ab7e58" exitCode=0
Jul 31 15:14:57.876542 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.876525    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"
Jul 31 15:14:57.877029 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.876536    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp" event={"ID":"1e2e6c85-a46e-4b3b-b4da-da97f0177a30","Type":"ContainerDied","Data":"d9863915e74707ae3d6a6bad8e049dc9b98e18ff61847295c32d8973b5ab7e58"}
Jul 31 15:14:57.877029 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.876605    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp" event={"ID":"1e2e6c85-a46e-4b3b-b4da-da97f0177a30","Type":"ContainerDied","Data":"cca071af5dcf76f6f7940e3292d9ff07f2d7de53953034acf9db07f0f3d65184"}
Jul 31 15:14:57.877029 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.876638    2578 scope.go:117] "RemoveContainer" containerID="d9863915e74707ae3d6a6bad8e049dc9b98e18ff61847295c32d8973b5ab7e58"
Jul 31 15:14:57.878297 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.878229    2578 generic.go:356] "Generic (PLEG): container finished" podID="4376a9b4-95bd-4fbd-998e-0649d0d6a323" containerID="72976a88503f015ed34724a828359446bcd6d2f6322067778bfa24af5f4937c1" exitCode=0
Jul 31 15:14:57.878436 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.878297    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"
Jul 31 15:14:57.878436 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.878302    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd" event={"ID":"4376a9b4-95bd-4fbd-998e-0649d0d6a323","Type":"ContainerDied","Data":"72976a88503f015ed34724a828359446bcd6d2f6322067778bfa24af5f4937c1"}
Jul 31 15:14:57.878436 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.878341    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd" event={"ID":"4376a9b4-95bd-4fbd-998e-0649d0d6a323","Type":"ContainerDied","Data":"6d44e3df0d58decf8c197b8958eadabf9c87ab492ed57b12a375aa1018e9a40f"}
Jul 31 15:14:57.886242 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.886220    2578 scope.go:117] "RemoveContainer" containerID="d07e3baf04378ada1ec563c2b56660e0f32e2b040ad80bf98e9455097b08f870"
Jul 31 15:14:57.893687 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.893669    2578 scope.go:117] "RemoveContainer" containerID="d9863915e74707ae3d6a6bad8e049dc9b98e18ff61847295c32d8973b5ab7e58"
Jul 31 15:14:57.893935 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:14:57.893916    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9863915e74707ae3d6a6bad8e049dc9b98e18ff61847295c32d8973b5ab7e58\": container with ID starting with d9863915e74707ae3d6a6bad8e049dc9b98e18ff61847295c32d8973b5ab7e58 not found: ID does not exist" containerID="d9863915e74707ae3d6a6bad8e049dc9b98e18ff61847295c32d8973b5ab7e58"
Jul 31 15:14:57.893982 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.893942    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9863915e74707ae3d6a6bad8e049dc9b98e18ff61847295c32d8973b5ab7e58"} err="failed to get container status \"d9863915e74707ae3d6a6bad8e049dc9b98e18ff61847295c32d8973b5ab7e58\": rpc error: code = NotFound desc = could not find container \"d9863915e74707ae3d6a6bad8e049dc9b98e18ff61847295c32d8973b5ab7e58\": container with ID starting with d9863915e74707ae3d6a6bad8e049dc9b98e18ff61847295c32d8973b5ab7e58 not found: ID does not exist"
Jul 31 15:14:57.893982 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.893958    2578 scope.go:117] "RemoveContainer" containerID="d07e3baf04378ada1ec563c2b56660e0f32e2b040ad80bf98e9455097b08f870"
Jul 31 15:14:57.894197 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:14:57.894180    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07e3baf04378ada1ec563c2b56660e0f32e2b040ad80bf98e9455097b08f870\": container with ID starting with d07e3baf04378ada1ec563c2b56660e0f32e2b040ad80bf98e9455097b08f870 not found: ID does not exist" containerID="d07e3baf04378ada1ec563c2b56660e0f32e2b040ad80bf98e9455097b08f870"
Jul 31 15:14:57.894258 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.894202    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07e3baf04378ada1ec563c2b56660e0f32e2b040ad80bf98e9455097b08f870"} err="failed to get container status \"d07e3baf04378ada1ec563c2b56660e0f32e2b040ad80bf98e9455097b08f870\": rpc error: code = NotFound desc = could not find container \"d07e3baf04378ada1ec563c2b56660e0f32e2b040ad80bf98e9455097b08f870\": container with ID starting with d07e3baf04378ada1ec563c2b56660e0f32e2b040ad80bf98e9455097b08f870 not found: ID does not exist"
Jul 31 15:14:57.894258 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.894215    2578 scope.go:117] "RemoveContainer" containerID="72976a88503f015ed34724a828359446bcd6d2f6322067778bfa24af5f4937c1"
Jul 31 15:14:57.901295 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.901277    2578 scope.go:117] "RemoveContainer" containerID="0e948b0ec3c3c264180366c8db938597fbdb20cb32ccefc35e1804f5c6a42654"
Jul 31 15:14:57.909727 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.909710    2578 scope.go:117] "RemoveContainer" containerID="72976a88503f015ed34724a828359446bcd6d2f6322067778bfa24af5f4937c1"
Jul 31 15:14:57.910045 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:14:57.910022    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72976a88503f015ed34724a828359446bcd6d2f6322067778bfa24af5f4937c1\": container with ID starting with 72976a88503f015ed34724a828359446bcd6d2f6322067778bfa24af5f4937c1 not found: ID does not exist" containerID="72976a88503f015ed34724a828359446bcd6d2f6322067778bfa24af5f4937c1"
Jul 31 15:14:57.910102 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.910053    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72976a88503f015ed34724a828359446bcd6d2f6322067778bfa24af5f4937c1"} err="failed to get container status \"72976a88503f015ed34724a828359446bcd6d2f6322067778bfa24af5f4937c1\": rpc error: code = NotFound desc = could not find container \"72976a88503f015ed34724a828359446bcd6d2f6322067778bfa24af5f4937c1\": container with ID starting with 72976a88503f015ed34724a828359446bcd6d2f6322067778bfa24af5f4937c1 not found: ID does not exist"
Jul 31 15:14:57.910102 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.910070    2578 scope.go:117] "RemoveContainer" containerID="0e948b0ec3c3c264180366c8db938597fbdb20cb32ccefc35e1804f5c6a42654"
Jul 31 15:14:57.910191 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.910090    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"]
Jul 31 15:14:57.910315 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:14:57.910300    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e948b0ec3c3c264180366c8db938597fbdb20cb32ccefc35e1804f5c6a42654\": container with ID starting with 0e948b0ec3c3c264180366c8db938597fbdb20cb32ccefc35e1804f5c6a42654 not found: ID does not exist" containerID="0e948b0ec3c3c264180366c8db938597fbdb20cb32ccefc35e1804f5c6a42654"
Jul 31 15:14:57.910388 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.910317    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e948b0ec3c3c264180366c8db938597fbdb20cb32ccefc35e1804f5c6a42654"} err="failed to get container status \"0e948b0ec3c3c264180366c8db938597fbdb20cb32ccefc35e1804f5c6a42654\": rpc error: code = NotFound desc = could not find container \"0e948b0ec3c3c264180366c8db938597fbdb20cb32ccefc35e1804f5c6a42654\": container with ID starting with 0e948b0ec3c3c264180366c8db938597fbdb20cb32ccefc35e1804f5c6a42654 not found: ID does not exist"
Jul 31 15:14:57.914779 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.914758    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-8g8bp"]
Jul 31 15:14:57.929759 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.929723    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"]
Jul 31 15:14:57.932930 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:57.932900    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b5959dc56-r5vmd"]
Jul 31 15:14:58.489320 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:58.489284    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e2e6c85-a46e-4b3b-b4da-da97f0177a30" path="/var/lib/kubelet/pods/1e2e6c85-a46e-4b3b-b4da-da97f0177a30/volumes"
Jul 31 15:14:58.489739 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:14:58.489725    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4376a9b4-95bd-4fbd-998e-0649d0d6a323" path="/var/lib/kubelet/pods/4376a9b4-95bd-4fbd-998e-0649d0d6a323/volumes"
Jul 31 15:15:11.492026 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.491989    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"]
Jul 31 15:15:11.492547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.492247    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4376a9b4-95bd-4fbd-998e-0649d0d6a323" containerName="storage-initializer"
Jul 31 15:15:11.492547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.492258    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4376a9b4-95bd-4fbd-998e-0649d0d6a323" containerName="storage-initializer"
Jul 31 15:15:11.492547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.492273    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc757396-4b7a-4959-8ba4-c987bb2b6b86" containerName="main"
Jul 31 15:15:11.492547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.492278    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc757396-4b7a-4959-8ba4-c987bb2b6b86" containerName="main"
Jul 31 15:15:11.492547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.492286    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4376a9b4-95bd-4fbd-998e-0649d0d6a323" containerName="main"
Jul 31 15:15:11.492547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.492291    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4376a9b4-95bd-4fbd-998e-0649d0d6a323" containerName="main"
Jul 31 15:15:11.492547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.492301    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2e6c85-a46e-4b3b-b4da-da97f0177a30" containerName="storage-initializer"
Jul 31 15:15:11.492547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.492307    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2e6c85-a46e-4b3b-b4da-da97f0177a30" containerName="storage-initializer"
Jul 31 15:15:11.492547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.492314    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fb9b0de-0a30-45bd-9beb-731f5197f830" containerName="main"
Jul 31 15:15:11.492547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.492319    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fb9b0de-0a30-45bd-9beb-731f5197f830" containerName="main"
Jul 31 15:15:11.492547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.492325    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2e6c85-a46e-4b3b-b4da-da97f0177a30" containerName="main"
Jul 31 15:15:11.492547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.492331    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2e6c85-a46e-4b3b-b4da-da97f0177a30" containerName="main"
Jul 31 15:15:11.492547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.492397    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc757396-4b7a-4959-8ba4-c987bb2b6b86" containerName="main"
Jul 31 15:15:11.492547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.492405    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4376a9b4-95bd-4fbd-998e-0649d0d6a323" containerName="main"
Jul 31 15:15:11.492547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.492412    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e2e6c85-a46e-4b3b-b4da-da97f0177a30" containerName="main"
Jul 31 15:15:11.492547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.492418    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fb9b0de-0a30-45bd-9beb-731f5197f830" containerName="main"
Jul 31 15:15:11.495297 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.495279    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.499084 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.499062    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-gfzr2\""
Jul 31 15:15:11.499232 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.499063    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Jul 31 15:15:11.499232 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.499100    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Jul 31 15:15:11.499232 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.499063    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Jul 31 15:15:11.509148 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.509125    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"]
Jul 31 15:15:11.594085 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.594052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-tls-certs\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.594085 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.594090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-model-cache\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.594385 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.594192    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.594385 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.594233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdwq5\" (UniqueName: \"kubernetes.io/projected/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-kube-api-access-xdwq5\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.594385 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.594280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-home\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.594385 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.594376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-dshm\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.695595 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.695547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.695802 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.695605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdwq5\" (UniqueName: \"kubernetes.io/projected/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-kube-api-access-xdwq5\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.695802 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.695642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-home\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.695802 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.695696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-dshm\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.695802 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.695743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-tls-certs\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.695802 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.695766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-model-cache\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.696072 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.696024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.696135 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.696068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-home\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.696135 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.696086    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-model-cache\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.698066 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.698045    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-dshm\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.698251 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.698232    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-tls-certs\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.703768 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.703746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdwq5\" (UniqueName: \"kubernetes.io/projected/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-kube-api-access-xdwq5\") pod \"test-div-v1-kserve-fb566d667-96cft\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.805728 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.805643    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:11.932999 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:11.932962    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"]
Jul 31 15:15:11.935940 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:15:11.935910    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2a4e0e_a5d6_40b6_ba18_7bc70c5fde67.slice/crio-4858b4a24e77548a2acc44aee890b421d08a65bda3d1db9857e31c1757e30770 WatchSource:0}: Error finding container 4858b4a24e77548a2acc44aee890b421d08a65bda3d1db9857e31c1757e30770: Status 404 returned error can't find the container with id 4858b4a24e77548a2acc44aee890b421d08a65bda3d1db9857e31c1757e30770
Jul 31 15:15:12.944669 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:12.944634    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft" event={"ID":"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67","Type":"ContainerStarted","Data":"cbefc967f2f35ce32f5f3f127e7559395441aaed826c171b7dea9fe5c9b70772"}
Jul 31 15:15:12.945047 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:12.944687    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft" event={"ID":"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67","Type":"ContainerStarted","Data":"4858b4a24e77548a2acc44aee890b421d08a65bda3d1db9857e31c1757e30770"}
Jul 31 15:15:12.963843 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:12.963800    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft" podStartSLOduration=1.963787205 podStartE2EDuration="1.963787205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:15:12.962769299 +0000 UTC m=+1610.971981187" watchObservedRunningTime="2026-07-31 15:15:12.963787205 +0000 UTC m=+1610.972999104"
Jul 31 15:15:13.806489 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:13.806431    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:21.805969 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:21.805890    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:21.818650 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:21.818627    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:21.991673 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:21.991646    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:15:46.113305 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.113265    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"]
Jul 31 15:15:46.116076 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.116057    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.120695 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.120665    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-h48tb\""
Jul 31 15:15:46.120951 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.120933    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 31 15:15:46.121964 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.121944    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 31 15:15:46.122362 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.122327    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:15:46.122737 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.122718    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 31 15:15:46.123070 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.123045    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 31 15:15:46.129837 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.129797    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"]
Jul 31 15:15:46.282982 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.282949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-home\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.282982 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.282989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gddvd\" (UniqueName: \"kubernetes.io/projected/1031427c-387d-4ad2-8691-96d126b24356-kube-api-access-gddvd\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.283235 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.283009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1031427c-387d-4ad2-8691-96d126b24356-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.283235 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.283028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1031427c-387d-4ad2-8691-96d126b24356-tls-certs\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.283235 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.283106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-model-cache\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.283235 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.283169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.283452 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.283234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-dshm\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.283452 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.283280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.384685 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.384609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-model-cache\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.384685 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.384664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.384892 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.384692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-dshm\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.384892 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.384713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.384892 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.384738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-home\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.384892 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.384770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gddvd\" (UniqueName: \"kubernetes.io/projected/1031427c-387d-4ad2-8691-96d126b24356-kube-api-access-gddvd\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.384892 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.384796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1031427c-387d-4ad2-8691-96d126b24356-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.384892 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.384819    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1031427c-387d-4ad2-8691-96d126b24356-tls-certs\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.385191 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.385007    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-model-cache\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.385191 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.385073    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.385300 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.385236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-home\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.385300 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.385237    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.385555 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.385535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1031427c-387d-4ad2-8691-96d126b24356-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.386991 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.386973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-dshm\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.387340 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.387325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1031427c-387d-4ad2-8691-96d126b24356-tls-certs\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.396719 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.396697    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gddvd\" (UniqueName: \"kubernetes.io/projected/1031427c-387d-4ad2-8691-96d126b24356-kube-api-access-gddvd\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.416135 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.416099    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"]
Jul 31 15:15:46.418717 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.418700    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:15:46.433166 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.433020    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"]
Jul 31 15:15:46.434631 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.434611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:46.567371 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.567311    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"]
Jul 31 15:15:46.570581 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:15:46.570550    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1031427c_387d_4ad2_8691_96d126b24356.slice/crio-91f9193fd28b764e8b6f2dade201563da89e48a8a9e7beeab65cfdd96bc22256 WatchSource:0}: Error finding container 91f9193fd28b764e8b6f2dade201563da89e48a8a9e7beeab65cfdd96bc22256: Status 404 returned error can't find the container with id 91f9193fd28b764e8b6f2dade201563da89e48a8a9e7beeab65cfdd96bc22256
Jul 31 15:15:46.586387 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.586345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f81b74fa-9706-476d-bb81-22da0a63fbcc-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5\" (UID: \"f81b74fa-9706-476d-bb81-22da0a63fbcc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:15:46.586486 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.586409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f81b74fa-9706-476d-bb81-22da0a63fbcc-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5\" (UID: \"f81b74fa-9706-476d-bb81-22da0a63fbcc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:15:46.586486 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.586430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f81b74fa-9706-476d-bb81-22da0a63fbcc-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5\" (UID: \"f81b74fa-9706-476d-bb81-22da0a63fbcc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:15:46.586486 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.586466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f81b74fa-9706-476d-bb81-22da0a63fbcc-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5\" (UID: \"f81b74fa-9706-476d-bb81-22da0a63fbcc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:15:46.687926 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.687828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f81b74fa-9706-476d-bb81-22da0a63fbcc-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5\" (UID: \"f81b74fa-9706-476d-bb81-22da0a63fbcc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:15:46.687926 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.687868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f81b74fa-9706-476d-bb81-22da0a63fbcc-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5\" (UID: \"f81b74fa-9706-476d-bb81-22da0a63fbcc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:15:46.687926 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.687889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f81b74fa-9706-476d-bb81-22da0a63fbcc-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5\" (UID: \"f81b74fa-9706-476d-bb81-22da0a63fbcc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:15:46.687926 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.687925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f81b74fa-9706-476d-bb81-22da0a63fbcc-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5\" (UID: \"f81b74fa-9706-476d-bb81-22da0a63fbcc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:15:46.688285 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.688233    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f81b74fa-9706-476d-bb81-22da0a63fbcc-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5\" (UID: \"f81b74fa-9706-476d-bb81-22da0a63fbcc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:15:46.688386 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.688340    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f81b74fa-9706-476d-bb81-22da0a63fbcc-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5\" (UID: \"f81b74fa-9706-476d-bb81-22da0a63fbcc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:15:46.688600 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.688579    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f81b74fa-9706-476d-bb81-22da0a63fbcc-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5\" (UID: \"f81b74fa-9706-476d-bb81-22da0a63fbcc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:15:46.690271 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.690248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f81b74fa-9706-476d-bb81-22da0a63fbcc-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5\" (UID: \"f81b74fa-9706-476d-bb81-22da0a63fbcc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:15:46.730314 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.730277    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:15:46.858947 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:46.858916    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"]
Jul 31 15:15:46.862419 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:15:46.862375    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81b74fa_9706_476d_bb81_22da0a63fbcc.slice/crio-dc06664e27725c99063f29820e6548379ce79d5506bc17a1e02d48b38827a649 WatchSource:0}: Error finding container dc06664e27725c99063f29820e6548379ce79d5506bc17a1e02d48b38827a649: Status 404 returned error can't find the container with id dc06664e27725c99063f29820e6548379ce79d5506bc17a1e02d48b38827a649
Jul 31 15:15:47.075790 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:47.075746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5" event={"ID":"f81b74fa-9706-476d-bb81-22da0a63fbcc","Type":"ContainerStarted","Data":"0f637ee7db2b468ab6ebd66dc6d46fc528f2a1c4b8403c9357437a8cf3b0b36b"}
Jul 31 15:15:47.075997 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:47.075814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5" event={"ID":"f81b74fa-9706-476d-bb81-22da0a63fbcc","Type":"ContainerStarted","Data":"dc06664e27725c99063f29820e6548379ce79d5506bc17a1e02d48b38827a649"}
Jul 31 15:15:47.077340 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:47.077313    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf" event={"ID":"1031427c-387d-4ad2-8691-96d126b24356","Type":"ContainerStarted","Data":"2d71c331814c8c67455754fffd89d5943b39718e275c04caad9aded7bad8ccf9"}
Jul 31 15:15:47.077494 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:47.077381    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf" event={"ID":"1031427c-387d-4ad2-8691-96d126b24356","Type":"ContainerStarted","Data":"91f9193fd28b764e8b6f2dade201563da89e48a8a9e7beeab65cfdd96bc22256"}
Jul 31 15:15:50.092139 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:50.092095    2578 generic.go:356] "Generic (PLEG): container finished" podID="f81b74fa-9706-476d-bb81-22da0a63fbcc" containerID="0f637ee7db2b468ab6ebd66dc6d46fc528f2a1c4b8403c9357437a8cf3b0b36b" exitCode=0
Jul 31 15:15:50.092597 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:50.092167    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5" event={"ID":"f81b74fa-9706-476d-bb81-22da0a63fbcc","Type":"ContainerDied","Data":"0f637ee7db2b468ab6ebd66dc6d46fc528f2a1c4b8403c9357437a8cf3b0b36b"}
Jul 31 15:15:50.093671 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:50.093647    2578 generic.go:356] "Generic (PLEG): container finished" podID="1031427c-387d-4ad2-8691-96d126b24356" containerID="2d71c331814c8c67455754fffd89d5943b39718e275c04caad9aded7bad8ccf9" exitCode=0
Jul 31 15:15:50.093797 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:50.093717    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf" event={"ID":"1031427c-387d-4ad2-8691-96d126b24356","Type":"ContainerDied","Data":"2d71c331814c8c67455754fffd89d5943b39718e275c04caad9aded7bad8ccf9"}
Jul 31 15:15:51.099308 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:51.099272    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5" event={"ID":"f81b74fa-9706-476d-bb81-22da0a63fbcc","Type":"ContainerStarted","Data":"e698338a191ef77dfa68d9fa461d786609bd736e75f39e73ccadc1f8ea5ce65b"}
Jul 31 15:15:51.099785 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:51.099529    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:15:51.101161 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:51.101137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf" event={"ID":"1031427c-387d-4ad2-8691-96d126b24356","Type":"ContainerStarted","Data":"e85fe0920449477fdc8ed3bf5086f5e55a82668e58611b0c5452177bda06af04"}
Jul 31 15:15:51.118488 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:51.118443    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5" podStartSLOduration=2.619653514 podStartE2EDuration="5.118428185s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.498774671s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:15:51.117449308 +0000 UTC m=+1649.126661177" watchObservedRunningTime="2026-07-31 15:15:51.118428185 +0000 UTC m=+1649.127640053"
Jul 31 15:15:51.137948 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:51.137904    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf" podStartSLOduration=2.651683631 podStartE2EDuration="5.137889907s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.486206276s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:15:51.137181203 +0000 UTC m=+1649.146393084" watchObservedRunningTime="2026-07-31 15:15:51.137889907 +0000 UTC m=+1649.147101775"
Jul 31 15:15:51.435878 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:51.435777    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:56.435555 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:56.435514    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:56.448447 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:56.448418    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:15:57.136689 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:15:57.136658    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:16:22.118341 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:22.118306    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:16:23.025068 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:23.025020    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5_f81b74fa-9706-476d-bb81-22da0a63fbcc/main/0.log"
Jul 31 15:16:23.453819 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:23.453781    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"]
Jul 31 15:16:23.464097 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:16:23.464062    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:16:23.464097 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:16:23.464097 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:16:23.464097 ip-10-0-133-32 kubenswrapper[2578]:  > containerID="e85fe0920449477fdc8ed3bf5086f5e55a82668e58611b0c5452177bda06af04" cmd=["/bin/sleep","15"]
Jul 31 15:16:23.464429 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:23.464128    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf" message="" err=<
Jul 31 15:16:23.464429 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:16:23.464429 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:16:23.464429 ip-10-0-133-32 kubenswrapper[2578]:  >
Jul 31 15:16:23.464429 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:16:23.464144    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:16:23.464429 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:16:23.464429 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:16:23.464429 ip-10-0-133-32 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf" podUID="1031427c-387d-4ad2-8691-96d126b24356" containerName="main" containerID="cri-o://e85fe0920449477fdc8ed3bf5086f5e55a82668e58611b0c5452177bda06af04"
Jul 31 15:16:23.464429 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:23.464196    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf" podUID="1031427c-387d-4ad2-8691-96d126b24356" containerName="main" containerID="cri-o://e85fe0920449477fdc8ed3bf5086f5e55a82668e58611b0c5452177bda06af04" gracePeriod=60
Jul 31 15:16:23.469952 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:23.469923    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"]
Jul 31 15:16:23.470155 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:23.470134    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5" podUID="f81b74fa-9706-476d-bb81-22da0a63fbcc" containerName="main" containerID="cri-o://e698338a191ef77dfa68d9fa461d786609bd736e75f39e73ccadc1f8ea5ce65b" gracePeriod=30
Jul 31 15:16:24.010148 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.010123    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:16:24.104656 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.103907    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f81b74fa-9706-476d-bb81-22da0a63fbcc-kserve-provision-location\") pod \"f81b74fa-9706-476d-bb81-22da0a63fbcc\" (UID: \"f81b74fa-9706-476d-bb81-22da0a63fbcc\") "
Jul 31 15:16:24.104656 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.104019    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f81b74fa-9706-476d-bb81-22da0a63fbcc-cabundle-cert\") pod \"f81b74fa-9706-476d-bb81-22da0a63fbcc\" (UID: \"f81b74fa-9706-476d-bb81-22da0a63fbcc\") "
Jul 31 15:16:24.104656 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.104058    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f81b74fa-9706-476d-bb81-22da0a63fbcc-tmp-dir\") pod \"f81b74fa-9706-476d-bb81-22da0a63fbcc\" (UID: \"f81b74fa-9706-476d-bb81-22da0a63fbcc\") "
Jul 31 15:16:24.104656 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.104093    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f81b74fa-9706-476d-bb81-22da0a63fbcc-tls-certs\") pod \"f81b74fa-9706-476d-bb81-22da0a63fbcc\" (UID: \"f81b74fa-9706-476d-bb81-22da0a63fbcc\") "
Jul 31 15:16:24.105466 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.105292    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81b74fa-9706-476d-bb81-22da0a63fbcc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f81b74fa-9706-476d-bb81-22da0a63fbcc" (UID: "f81b74fa-9706-476d-bb81-22da0a63fbcc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:16:24.105595 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.105523    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81b74fa-9706-476d-bb81-22da0a63fbcc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f81b74fa-9706-476d-bb81-22da0a63fbcc" (UID: "f81b74fa-9706-476d-bb81-22da0a63fbcc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:24.107476 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.107447    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f81b74fa-9706-476d-bb81-22da0a63fbcc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f81b74fa-9706-476d-bb81-22da0a63fbcc" (UID: "f81b74fa-9706-476d-bb81-22da0a63fbcc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:16:24.125982 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.125940    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf" podUID="1031427c-387d-4ad2-8691-96d126b24356" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 31 15:16:24.130233 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.128708    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81b74fa-9706-476d-bb81-22da0a63fbcc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f81b74fa-9706-476d-bb81-22da0a63fbcc" (UID: "f81b74fa-9706-476d-bb81-22da0a63fbcc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:24.205113 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.205087    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f81b74fa-9706-476d-bb81-22da0a63fbcc-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:24.205113 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.205112    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f81b74fa-9706-476d-bb81-22da0a63fbcc-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:24.205300 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.205122    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f81b74fa-9706-476d-bb81-22da0a63fbcc-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:24.205300 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.205132    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f81b74fa-9706-476d-bb81-22da0a63fbcc-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:24.208320 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.208298    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:16:24.242619 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.242591    2578 generic.go:356] "Generic (PLEG): container finished" podID="f81b74fa-9706-476d-bb81-22da0a63fbcc" containerID="e698338a191ef77dfa68d9fa461d786609bd736e75f39e73ccadc1f8ea5ce65b" exitCode=0
Jul 31 15:16:24.242783 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.242641    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"
Jul 31 15:16:24.242783 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.242673    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5" event={"ID":"f81b74fa-9706-476d-bb81-22da0a63fbcc","Type":"ContainerDied","Data":"e698338a191ef77dfa68d9fa461d786609bd736e75f39e73ccadc1f8ea5ce65b"}
Jul 31 15:16:24.242783 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.242763    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5" event={"ID":"f81b74fa-9706-476d-bb81-22da0a63fbcc","Type":"ContainerDied","Data":"dc06664e27725c99063f29820e6548379ce79d5506bc17a1e02d48b38827a649"}
Jul 31 15:16:24.242968 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.242794    2578 scope.go:117] "RemoveContainer" containerID="e698338a191ef77dfa68d9fa461d786609bd736e75f39e73ccadc1f8ea5ce65b"
Jul 31 15:16:24.247766 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.247737    2578 generic.go:356] "Generic (PLEG): container finished" podID="1031427c-387d-4ad2-8691-96d126b24356" containerID="e85fe0920449477fdc8ed3bf5086f5e55a82668e58611b0c5452177bda06af04" exitCode=0
Jul 31 15:16:24.247908 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.247813    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"
Jul 31 15:16:24.247908 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.247817    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf" event={"ID":"1031427c-387d-4ad2-8691-96d126b24356","Type":"ContainerDied","Data":"e85fe0920449477fdc8ed3bf5086f5e55a82668e58611b0c5452177bda06af04"}
Jul 31 15:16:24.247908 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.247871    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf" event={"ID":"1031427c-387d-4ad2-8691-96d126b24356","Type":"ContainerDied","Data":"91f9193fd28b764e8b6f2dade201563da89e48a8a9e7beeab65cfdd96bc22256"}
Jul 31 15:16:24.254896 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.254879    2578 scope.go:117] "RemoveContainer" containerID="0f637ee7db2b468ab6ebd66dc6d46fc528f2a1c4b8403c9357437a8cf3b0b36b"
Jul 31 15:16:24.262077 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.262058    2578 scope.go:117] "RemoveContainer" containerID="e698338a191ef77dfa68d9fa461d786609bd736e75f39e73ccadc1f8ea5ce65b"
Jul 31 15:16:24.262296 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:16:24.262280    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e698338a191ef77dfa68d9fa461d786609bd736e75f39e73ccadc1f8ea5ce65b\": container with ID starting with e698338a191ef77dfa68d9fa461d786609bd736e75f39e73ccadc1f8ea5ce65b not found: ID does not exist" containerID="e698338a191ef77dfa68d9fa461d786609bd736e75f39e73ccadc1f8ea5ce65b"
Jul 31 15:16:24.262365 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.262303    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e698338a191ef77dfa68d9fa461d786609bd736e75f39e73ccadc1f8ea5ce65b"} err="failed to get container status \"e698338a191ef77dfa68d9fa461d786609bd736e75f39e73ccadc1f8ea5ce65b\": rpc error: code = NotFound desc = could not find container \"e698338a191ef77dfa68d9fa461d786609bd736e75f39e73ccadc1f8ea5ce65b\": container with ID starting with e698338a191ef77dfa68d9fa461d786609bd736e75f39e73ccadc1f8ea5ce65b not found: ID does not exist"
Jul 31 15:16:24.262365 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.262318    2578 scope.go:117] "RemoveContainer" containerID="0f637ee7db2b468ab6ebd66dc6d46fc528f2a1c4b8403c9357437a8cf3b0b36b"
Jul 31 15:16:24.262553 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:16:24.262534    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f637ee7db2b468ab6ebd66dc6d46fc528f2a1c4b8403c9357437a8cf3b0b36b\": container with ID starting with 0f637ee7db2b468ab6ebd66dc6d46fc528f2a1c4b8403c9357437a8cf3b0b36b not found: ID does not exist" containerID="0f637ee7db2b468ab6ebd66dc6d46fc528f2a1c4b8403c9357437a8cf3b0b36b"
Jul 31 15:16:24.262595 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.262561    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f637ee7db2b468ab6ebd66dc6d46fc528f2a1c4b8403c9357437a8cf3b0b36b"} err="failed to get container status \"0f637ee7db2b468ab6ebd66dc6d46fc528f2a1c4b8403c9357437a8cf3b0b36b\": rpc error: code = NotFound desc = could not find container \"0f637ee7db2b468ab6ebd66dc6d46fc528f2a1c4b8403c9357437a8cf3b0b36b\": container with ID starting with 0f637ee7db2b468ab6ebd66dc6d46fc528f2a1c4b8403c9357437a8cf3b0b36b not found: ID does not exist"
Jul 31 15:16:24.262595 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.262577    2578 scope.go:117] "RemoveContainer" containerID="e85fe0920449477fdc8ed3bf5086f5e55a82668e58611b0c5452177bda06af04"
Jul 31 15:16:24.266928 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.266909    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"]
Jul 31 15:16:24.270197 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.270176    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-77db79d6db-tppt5"]
Jul 31 15:16:24.271142 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.271130    2578 scope.go:117] "RemoveContainer" containerID="2d71c331814c8c67455754fffd89d5943b39718e275c04caad9aded7bad8ccf9"
Jul 31 15:16:24.277948 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.277934    2578 scope.go:117] "RemoveContainer" containerID="e85fe0920449477fdc8ed3bf5086f5e55a82668e58611b0c5452177bda06af04"
Jul 31 15:16:24.278194 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:16:24.278177    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e85fe0920449477fdc8ed3bf5086f5e55a82668e58611b0c5452177bda06af04\": container with ID starting with e85fe0920449477fdc8ed3bf5086f5e55a82668e58611b0c5452177bda06af04 not found: ID does not exist" containerID="e85fe0920449477fdc8ed3bf5086f5e55a82668e58611b0c5452177bda06af04"
Jul 31 15:16:24.278245 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.278201    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e85fe0920449477fdc8ed3bf5086f5e55a82668e58611b0c5452177bda06af04"} err="failed to get container status \"e85fe0920449477fdc8ed3bf5086f5e55a82668e58611b0c5452177bda06af04\": rpc error: code = NotFound desc = could not find container \"e85fe0920449477fdc8ed3bf5086f5e55a82668e58611b0c5452177bda06af04\": container with ID starting with e85fe0920449477fdc8ed3bf5086f5e55a82668e58611b0c5452177bda06af04 not found: ID does not exist"
Jul 31 15:16:24.278245 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.278218    2578 scope.go:117] "RemoveContainer" containerID="2d71c331814c8c67455754fffd89d5943b39718e275c04caad9aded7bad8ccf9"
Jul 31 15:16:24.278501 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:16:24.278481    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d71c331814c8c67455754fffd89d5943b39718e275c04caad9aded7bad8ccf9\": container with ID starting with 2d71c331814c8c67455754fffd89d5943b39718e275c04caad9aded7bad8ccf9 not found: ID does not exist" containerID="2d71c331814c8c67455754fffd89d5943b39718e275c04caad9aded7bad8ccf9"
Jul 31 15:16:24.278557 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.278506    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d71c331814c8c67455754fffd89d5943b39718e275c04caad9aded7bad8ccf9"} err="failed to get container status \"2d71c331814c8c67455754fffd89d5943b39718e275c04caad9aded7bad8ccf9\": rpc error: code = NotFound desc = could not find container \"2d71c331814c8c67455754fffd89d5943b39718e275c04caad9aded7bad8ccf9\": container with ID starting with 2d71c331814c8c67455754fffd89d5943b39718e275c04caad9aded7bad8ccf9 not found: ID does not exist"
Jul 31 15:16:24.305943 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.305919    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-kserve-provision-location\") pod \"1031427c-387d-4ad2-8691-96d126b24356\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") "
Jul 31 15:16:24.306042 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.305952    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-dshm\") pod \"1031427c-387d-4ad2-8691-96d126b24356\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") "
Jul 31 15:16:24.306042 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.305983    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gddvd\" (UniqueName: \"kubernetes.io/projected/1031427c-387d-4ad2-8691-96d126b24356-kube-api-access-gddvd\") pod \"1031427c-387d-4ad2-8691-96d126b24356\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") "
Jul 31 15:16:24.306042 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.306032    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1031427c-387d-4ad2-8691-96d126b24356-cabundle-cert\") pod \"1031427c-387d-4ad2-8691-96d126b24356\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") "
Jul 31 15:16:24.306208 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.306068    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-model-cache\") pod \"1031427c-387d-4ad2-8691-96d126b24356\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") "
Jul 31 15:16:24.306208 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.306095    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-home\") pod \"1031427c-387d-4ad2-8691-96d126b24356\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") "
Jul 31 15:16:24.306208 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.306124    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1031427c-387d-4ad2-8691-96d126b24356-tls-certs\") pod \"1031427c-387d-4ad2-8691-96d126b24356\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") "
Jul 31 15:16:24.306208 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.306190    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-tmp-dir\") pod \"1031427c-387d-4ad2-8691-96d126b24356\" (UID: \"1031427c-387d-4ad2-8691-96d126b24356\") "
Jul 31 15:16:24.306420 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.306374    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-model-cache" (OuterVolumeSpecName: "model-cache") pod "1031427c-387d-4ad2-8691-96d126b24356" (UID: "1031427c-387d-4ad2-8691-96d126b24356"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:24.306478 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.306423    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-home" (OuterVolumeSpecName: "home") pod "1031427c-387d-4ad2-8691-96d126b24356" (UID: "1031427c-387d-4ad2-8691-96d126b24356"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:24.306478 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.306432    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1031427c-387d-4ad2-8691-96d126b24356-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1031427c-387d-4ad2-8691-96d126b24356" (UID: "1031427c-387d-4ad2-8691-96d126b24356"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:16:24.306625 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.306606    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1031427c-387d-4ad2-8691-96d126b24356" (UID: "1031427c-387d-4ad2-8691-96d126b24356"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:24.308078 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.308052    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1031427c-387d-4ad2-8691-96d126b24356-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1031427c-387d-4ad2-8691-96d126b24356" (UID: "1031427c-387d-4ad2-8691-96d126b24356"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:16:24.308180 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.308137    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-dshm" (OuterVolumeSpecName: "dshm") pod "1031427c-387d-4ad2-8691-96d126b24356" (UID: "1031427c-387d-4ad2-8691-96d126b24356"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:24.308296 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.308282    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1031427c-387d-4ad2-8691-96d126b24356-kube-api-access-gddvd" (OuterVolumeSpecName: "kube-api-access-gddvd") pod "1031427c-387d-4ad2-8691-96d126b24356" (UID: "1031427c-387d-4ad2-8691-96d126b24356"). InnerVolumeSpecName "kube-api-access-gddvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:16:24.327505 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.327453    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1031427c-387d-4ad2-8691-96d126b24356" (UID: "1031427c-387d-4ad2-8691-96d126b24356"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:24.407149 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.407111    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:24.407149 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.407142    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:24.407149 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.407156    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1031427c-387d-4ad2-8691-96d126b24356-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:24.407433 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.407166    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:24.407433 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.407178    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:24.407433 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.407188    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1031427c-387d-4ad2-8691-96d126b24356-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:24.407433 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.407200    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gddvd\" (UniqueName: \"kubernetes.io/projected/1031427c-387d-4ad2-8691-96d126b24356-kube-api-access-gddvd\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:24.407433 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.407211    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1031427c-387d-4ad2-8691-96d126b24356-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:24.489027 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.488991    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f81b74fa-9706-476d-bb81-22da0a63fbcc" path="/var/lib/kubelet/pods/f81b74fa-9706-476d-bb81-22da0a63fbcc/volumes"
Jul 31 15:16:24.565086 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.565055    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"]
Jul 31 15:16:24.571675 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:24.571646    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-8tqmf"]
Jul 31 15:16:26.488969 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:26.488935    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1031427c-387d-4ad2-8691-96d126b24356" path="/var/lib/kubelet/pods/1031427c-387d-4ad2-8691-96d126b24356/volumes"
Jul 31 15:16:30.813027 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:30.812989    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"]
Jul 31 15:16:31.826113 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.826079    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"]
Jul 31 15:16:31.826492 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.826384    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1031427c-387d-4ad2-8691-96d126b24356" containerName="main"
Jul 31 15:16:31.826492 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.826396    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1031427c-387d-4ad2-8691-96d126b24356" containerName="main"
Jul 31 15:16:31.826492 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.826412    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f81b74fa-9706-476d-bb81-22da0a63fbcc" containerName="main"
Jul 31 15:16:31.826492 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.826420    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f81b74fa-9706-476d-bb81-22da0a63fbcc" containerName="main"
Jul 31 15:16:31.826492 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.826431    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f81b74fa-9706-476d-bb81-22da0a63fbcc" containerName="storage-initializer"
Jul 31 15:16:31.826492 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.826438    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f81b74fa-9706-476d-bb81-22da0a63fbcc" containerName="storage-initializer"
Jul 31 15:16:31.826492 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.826444    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1031427c-387d-4ad2-8691-96d126b24356" containerName="storage-initializer"
Jul 31 15:16:31.826492 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.826451    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1031427c-387d-4ad2-8691-96d126b24356" containerName="storage-initializer"
Jul 31 15:16:31.826763 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.826503    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f81b74fa-9706-476d-bb81-22da0a63fbcc" containerName="main"
Jul 31 15:16:31.826763 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.826512    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1031427c-387d-4ad2-8691-96d126b24356" containerName="main"
Jul 31 15:16:31.829085 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.829066    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:31.831965 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.831940    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Jul 31 15:16:31.833261 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.833238    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-jz88b\""
Jul 31 15:16:31.833405 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.833283    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Jul 31 15:16:31.833405 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.833308    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Jul 31 15:16:31.842943 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.842920    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"]
Jul 31 15:16:31.970676 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.970638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f84fe81-97e0-4179-8613-60d959af937a-tls-certs\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:31.970676 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.970679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-dshm\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:31.970879 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.970706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-model-cache\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:31.970879 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.970756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:31.970879 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.970773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-home\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:31.970879 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:31.970864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52rc8\" (UniqueName: \"kubernetes.io/projected/4f84fe81-97e0-4179-8613-60d959af937a-kube-api-access-52rc8\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:32.072163 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:32.072126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52rc8\" (UniqueName: \"kubernetes.io/projected/4f84fe81-97e0-4179-8613-60d959af937a-kube-api-access-52rc8\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:32.072386 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:32.072173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f84fe81-97e0-4179-8613-60d959af937a-tls-certs\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:32.072386 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:32.072195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-dshm\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:32.072386 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:32.072220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-model-cache\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:32.072386 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:32.072243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:32.072386 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:32.072269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-home\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:32.072690 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:32.072659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-model-cache\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:32.072809 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:32.072696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-home\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:32.072809 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:32.072725    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:32.074485 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:32.074463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-dshm\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:32.074760 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:32.074741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f84fe81-97e0-4179-8613-60d959af937a-tls-certs\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:32.081150 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:32.081099    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52rc8\" (UniqueName: \"kubernetes.io/projected/4f84fe81-97e0-4179-8613-60d959af937a-kube-api-access-52rc8\") pod \"stop-v1-kserve-5657b88c95-zbb9s\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:32.139128 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:32.139088    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:32.270966 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:32.270939    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"]
Jul 31 15:16:32.273186 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:16:32.273158    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f84fe81_97e0_4179_8613_60d959af937a.slice/crio-9cb63cb6513ed793dad5a0df056b236b049de482f77df0d874343b5df06f2d12 WatchSource:0}: Error finding container 9cb63cb6513ed793dad5a0df056b236b049de482f77df0d874343b5df06f2d12: Status 404 returned error can't find the container with id 9cb63cb6513ed793dad5a0df056b236b049de482f77df0d874343b5df06f2d12
Jul 31 15:16:32.281959 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:32.281935    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s" event={"ID":"4f84fe81-97e0-4179-8613-60d959af937a","Type":"ContainerStarted","Data":"9cb63cb6513ed793dad5a0df056b236b049de482f77df0d874343b5df06f2d12"}
Jul 31 15:16:33.287406 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:33.287367    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s" event={"ID":"4f84fe81-97e0-4179-8613-60d959af937a","Type":"ContainerStarted","Data":"717c66d126423d56c52d610d27ee85ad9c85e536b3469fd2c0456aa383d15c1e"}
Jul 31 15:16:33.308093 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:33.308049    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s" podStartSLOduration=2.3080342480000002 podStartE2EDuration="2.308034248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:16:33.307330076 +0000 UTC m=+1691.316541945" watchObservedRunningTime="2026-07-31 15:16:33.308034248 +0000 UTC m=+1691.317246118"
Jul 31 15:16:34.139389 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:34.139336    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:37.002027 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.001992    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"]
Jul 31 15:16:37.004129 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.004104    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.006905 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.006882    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Jul 31 15:16:37.016702 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.016664    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"]
Jul 31 15:16:37.115154 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.115114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-dshm\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.115393 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.115171    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-model-cache\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.115393 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.115269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-tls-certs\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.115393 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.115307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2lcj\" (UniqueName: \"kubernetes.io/projected/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-kube-api-access-m2lcj\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.115560 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.115396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.115560 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.115431    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-home\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.216076 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.216039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-home\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.216281 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.216085    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-dshm\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.216281 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.216149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-model-cache\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.216445 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.216293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-tls-certs\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.216445 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.216343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2lcj\" (UniqueName: \"kubernetes.io/projected/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-kube-api-access-m2lcj\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.216445 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.216418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.216614 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.216512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-home\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.216614 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.216579    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-model-cache\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.216710 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.216692    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.218536 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.218509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-dshm\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.218785 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.218765    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-tls-certs\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.226117 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.226089    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2lcj\" (UniqueName: \"kubernetes.io/projected/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-kube-api-access-m2lcj\") pod \"stop-v2-kserve-7f68747c45-kmxq2\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.313871 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.313790    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:37.440799 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:37.440765    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"]
Jul 31 15:16:37.444121 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:16:37.444092    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca863ff7_b2b1_4c03_a3b8_8a9bb05396a0.slice/crio-8de26223191e67a80c42fb05a4b3331bab47b0d07a3b7e30d854494bfc44072b WatchSource:0}: Error finding container 8de26223191e67a80c42fb05a4b3331bab47b0d07a3b7e30d854494bfc44072b: Status 404 returned error can't find the container with id 8de26223191e67a80c42fb05a4b3331bab47b0d07a3b7e30d854494bfc44072b
Jul 31 15:16:38.312915 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:38.312877    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2" event={"ID":"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0","Type":"ContainerStarted","Data":"b575ed03266d69b2f1ad59d33723f8ba65a3621ef1d91c2d3d667c5f3bd19523"}
Jul 31 15:16:38.313335 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:38.312940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2" event={"ID":"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0","Type":"ContainerStarted","Data":"8de26223191e67a80c42fb05a4b3331bab47b0d07a3b7e30d854494bfc44072b"}
Jul 31 15:16:38.314217 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:38.314197    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:38.335784 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:38.335711    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2" podStartSLOduration=2.335688925 podStartE2EDuration="2.335688925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:16:38.333698846 +0000 UTC m=+1696.342910727" watchObservedRunningTime="2026-07-31 15:16:38.335688925 +0000 UTC m=+1696.344900797"
Jul 31 15:16:42.139972 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:42.139921    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:42.152770 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:42.152740    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:42.340882 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:42.340852    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:16:45.832068 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:45.831950    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft" podUID="1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67" containerName="main" containerID="cri-o://cbefc967f2f35ce32f5f3f127e7559395441aaed826c171b7dea9fe5c9b70772" gracePeriod=45
Jul 31 15:16:45.981362 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:45.981299    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft" podUID="1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 31 15:16:46.067836 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.067811    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:16:46.191791 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.191692    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-tmp-dir\") pod \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") "
Jul 31 15:16:46.191791 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.191744    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdwq5\" (UniqueName: \"kubernetes.io/projected/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-kube-api-access-xdwq5\") pod \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") "
Jul 31 15:16:46.191791 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.191769    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-model-cache\") pod \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") "
Jul 31 15:16:46.191791 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.191793    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-home\") pod \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") "
Jul 31 15:16:46.192134 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.191866    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-tls-certs\") pod \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") "
Jul 31 15:16:46.192134 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.191902    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-dshm\") pod \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\" (UID: \"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67\") "
Jul 31 15:16:46.192134 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.192046    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67" (UID: "1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:46.192134 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.192063    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67" (UID: "1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:46.192134 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.192088    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-home" (OuterVolumeSpecName: "home") pod "1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67" (UID: "1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:46.192452 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.192184    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:46.192452 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.192238    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:46.192452 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.192251    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:46.194195 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.194163    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-dshm" (OuterVolumeSpecName: "dshm") pod "1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67" (UID: "1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:46.194195 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.194179    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-kube-api-access-xdwq5" (OuterVolumeSpecName: "kube-api-access-xdwq5") pod "1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67" (UID: "1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67"). InnerVolumeSpecName "kube-api-access-xdwq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:16:46.194195 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.194183    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67" (UID: "1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:16:46.293345 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.293306    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:46.293345 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.293341    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:46.293345 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.293362    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdwq5\" (UniqueName: \"kubernetes.io/projected/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67-kube-api-access-xdwq5\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:16:46.345902 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.345863    2578 generic.go:356] "Generic (PLEG): container finished" podID="1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67" containerID="cbefc967f2f35ce32f5f3f127e7559395441aaed826c171b7dea9fe5c9b70772" exitCode=0
Jul 31 15:16:46.346064 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.345912    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"
Jul 31 15:16:46.346064 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.345951    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft" event={"ID":"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67","Type":"ContainerDied","Data":"cbefc967f2f35ce32f5f3f127e7559395441aaed826c171b7dea9fe5c9b70772"}
Jul 31 15:16:46.346064 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.346039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft" event={"ID":"1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67","Type":"ContainerDied","Data":"4858b4a24e77548a2acc44aee890b421d08a65bda3d1db9857e31c1757e30770"}
Jul 31 15:16:46.346256 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.346070    2578 scope.go:117] "RemoveContainer" containerID="cbefc967f2f35ce32f5f3f127e7559395441aaed826c171b7dea9fe5c9b70772"
Jul 31 15:16:46.354703 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.354686    2578 scope.go:117] "RemoveContainer" containerID="cbefc967f2f35ce32f5f3f127e7559395441aaed826c171b7dea9fe5c9b70772"
Jul 31 15:16:46.354974 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:16:46.354947    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbefc967f2f35ce32f5f3f127e7559395441aaed826c171b7dea9fe5c9b70772\": container with ID starting with cbefc967f2f35ce32f5f3f127e7559395441aaed826c171b7dea9fe5c9b70772 not found: ID does not exist" containerID="cbefc967f2f35ce32f5f3f127e7559395441aaed826c171b7dea9fe5c9b70772"
Jul 31 15:16:46.355049 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.354980    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbefc967f2f35ce32f5f3f127e7559395441aaed826c171b7dea9fe5c9b70772"} err="failed to get container status \"cbefc967f2f35ce32f5f3f127e7559395441aaed826c171b7dea9fe5c9b70772\": rpc error: code = NotFound desc = could not find container \"cbefc967f2f35ce32f5f3f127e7559395441aaed826c171b7dea9fe5c9b70772\": container with ID starting with cbefc967f2f35ce32f5f3f127e7559395441aaed826c171b7dea9fe5c9b70772 not found: ID does not exist"
Jul 31 15:16:46.368188 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.368166    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"]
Jul 31 15:16:46.371962 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.371943    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-96cft"]
Jul 31 15:16:46.492551 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:46.492508    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67" path="/var/lib/kubelet/pods/1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67/volumes"
Jul 31 15:16:47.314303 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:47.314267    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:47.326685 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:47.326659    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:16:47.361881 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:16:47.361857    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:17:16.605486 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.605448    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"]
Jul 31 15:17:16.605886 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.605709    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67" containerName="main"
Jul 31 15:17:16.605886 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.605719    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67" containerName="main"
Jul 31 15:17:16.605886 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.605794    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c2a4e0e-a5d6-40b6-ba18-7bc70c5fde67" containerName="main"
Jul 31 15:17:16.607629 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.607611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"
Jul 31 15:17:16.611625 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.611605    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-pkpd8\""
Jul 31 15:17:16.611744 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.611638    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 31 15:17:16.611975 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.611947    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 31 15:17:16.612054 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.611976    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 31 15:17:16.612054 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.611969    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-pkbzj\""
Jul 31 15:17:16.618662 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.618640    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"]
Jul 31 15:17:16.624643 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.624622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk6bx\" (UniqueName: \"kubernetes.io/projected/122713a8-44de-40f7-b8ed-89799c539df3-kube-api-access-fk6bx\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm\" (UID: \"122713a8-44de-40f7-b8ed-89799c539df3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"
Jul 31 15:17:16.624740 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.624656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/122713a8-44de-40f7-b8ed-89799c539df3-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm\" (UID: \"122713a8-44de-40f7-b8ed-89799c539df3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"
Jul 31 15:17:16.725017 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.724982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/122713a8-44de-40f7-b8ed-89799c539df3-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm\" (UID: \"122713a8-44de-40f7-b8ed-89799c539df3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"
Jul 31 15:17:16.725184 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.725069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk6bx\" (UniqueName: \"kubernetes.io/projected/122713a8-44de-40f7-b8ed-89799c539df3-kube-api-access-fk6bx\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm\" (UID: \"122713a8-44de-40f7-b8ed-89799c539df3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"
Jul 31 15:17:16.727444 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.727419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/122713a8-44de-40f7-b8ed-89799c539df3-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm\" (UID: \"122713a8-44de-40f7-b8ed-89799c539df3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"
Jul 31 15:17:16.733593 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.733564    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk6bx\" (UniqueName: \"kubernetes.io/projected/122713a8-44de-40f7-b8ed-89799c539df3-kube-api-access-fk6bx\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm\" (UID: \"122713a8-44de-40f7-b8ed-89799c539df3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"
Jul 31 15:17:16.918064 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:16.917970    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"
Jul 31 15:17:17.054481 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:17.054452    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"]
Jul 31 15:17:17.057200 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:17:17.057171    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122713a8_44de_40f7_b8ed_89799c539df3.slice/crio-2e3d01d2da66239a2678a901d33a402dc77d900c8a41145fc390c4e5010a5d8e WatchSource:0}: Error finding container 2e3d01d2da66239a2678a901d33a402dc77d900c8a41145fc390c4e5010a5d8e: Status 404 returned error can't find the container with id 2e3d01d2da66239a2678a901d33a402dc77d900c8a41145fc390c4e5010a5d8e
Jul 31 15:17:17.469536 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:17.469492    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm" event={"ID":"122713a8-44de-40f7-b8ed-89799c539df3","Type":"ContainerStarted","Data":"08dbaad1d65f64900710385b2ef89188e8f771e052941f60988244f441401b09"}
Jul 31 15:17:17.469728 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:17.469563    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm" event={"ID":"122713a8-44de-40f7-b8ed-89799c539df3","Type":"ContainerStarted","Data":"2e3d01d2da66239a2678a901d33a402dc77d900c8a41145fc390c4e5010a5d8e"}
Jul 31 15:17:17.469728 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:17.469596    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"
Jul 31 15:17:17.500937 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:17.500883    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm" podStartSLOduration=1.500865326 podStartE2EDuration="1.500865326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:17:17.494780078 +0000 UTC m=+1735.503991947" watchObservedRunningTime="2026-07-31 15:17:17.500865326 +0000 UTC m=+1735.510077194"
Jul 31 15:17:30.087545 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:30.087509    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"]
Jul 31 15:17:45.106500 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.106453    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2" podUID="ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0" containerName="main" containerID="cri-o://b575ed03266d69b2f1ad59d33723f8ba65a3621ef1d91c2d3d667c5f3bd19523" gracePeriod=45
Jul 31 15:17:45.345499 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.345474    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:17:45.465152 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.465064    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-model-cache\") pod \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") "
Jul 31 15:17:45.465152 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.465128    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-tmp-dir\") pod \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") "
Jul 31 15:17:45.465388 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.465160    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-dshm\") pod \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") "
Jul 31 15:17:45.465388 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.465181    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-tls-certs\") pod \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") "
Jul 31 15:17:45.465388 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.465201    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-home\") pod \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") "
Jul 31 15:17:45.465388 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.465241    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2lcj\" (UniqueName: \"kubernetes.io/projected/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-kube-api-access-m2lcj\") pod \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\" (UID: \"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0\") "
Jul 31 15:17:45.465598 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.465417    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0" (UID: "ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:17:45.465598 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.465447    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0" (UID: "ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:17:45.465598 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.465517    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-home" (OuterVolumeSpecName: "home") pod "ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0" (UID: "ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:17:45.467404 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.467372    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-dshm" (OuterVolumeSpecName: "dshm") pod "ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0" (UID: "ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:17:45.467524 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.467413    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0" (UID: "ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:17:45.467524 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.467459    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-kube-api-access-m2lcj" (OuterVolumeSpecName: "kube-api-access-m2lcj") pod "ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0" (UID: "ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0"). InnerVolumeSpecName "kube-api-access-m2lcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:17:45.566009 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.565975    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2lcj\" (UniqueName: \"kubernetes.io/projected/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-kube-api-access-m2lcj\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:17:45.566009 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.566005    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:17:45.566009 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.566015    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:17:45.566242 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.566023    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:17:45.566242 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.566031    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:17:45.566242 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.566038    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:17:45.582924 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.582885    2578 generic.go:356] "Generic (PLEG): container finished" podID="ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0" containerID="b575ed03266d69b2f1ad59d33723f8ba65a3621ef1d91c2d3d667c5f3bd19523" exitCode=0
Jul 31 15:17:45.583070 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.582930    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"
Jul 31 15:17:45.583070 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.582976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2" event={"ID":"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0","Type":"ContainerDied","Data":"b575ed03266d69b2f1ad59d33723f8ba65a3621ef1d91c2d3d667c5f3bd19523"}
Jul 31 15:17:45.583070 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.583025    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2" event={"ID":"ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0","Type":"ContainerDied","Data":"8de26223191e67a80c42fb05a4b3331bab47b0d07a3b7e30d854494bfc44072b"}
Jul 31 15:17:45.583070 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.583046    2578 scope.go:117] "RemoveContainer" containerID="b575ed03266d69b2f1ad59d33723f8ba65a3621ef1d91c2d3d667c5f3bd19523"
Jul 31 15:17:45.591760 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.591745    2578 scope.go:117] "RemoveContainer" containerID="b575ed03266d69b2f1ad59d33723f8ba65a3621ef1d91c2d3d667c5f3bd19523"
Jul 31 15:17:45.591989 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:17:45.591972    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b575ed03266d69b2f1ad59d33723f8ba65a3621ef1d91c2d3d667c5f3bd19523\": container with ID starting with b575ed03266d69b2f1ad59d33723f8ba65a3621ef1d91c2d3d667c5f3bd19523 not found: ID does not exist" containerID="b575ed03266d69b2f1ad59d33723f8ba65a3621ef1d91c2d3d667c5f3bd19523"
Jul 31 15:17:45.592040 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.591996    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b575ed03266d69b2f1ad59d33723f8ba65a3621ef1d91c2d3d667c5f3bd19523"} err="failed to get container status \"b575ed03266d69b2f1ad59d33723f8ba65a3621ef1d91c2d3d667c5f3bd19523\": rpc error: code = NotFound desc = could not find container \"b575ed03266d69b2f1ad59d33723f8ba65a3621ef1d91c2d3d667c5f3bd19523\": container with ID starting with b575ed03266d69b2f1ad59d33723f8ba65a3621ef1d91c2d3d667c5f3bd19523 not found: ID does not exist"
Jul 31 15:17:45.605410 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.605387    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"]
Jul 31 15:17:45.611444 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:45.611421    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-kmxq2"]
Jul 31 15:17:46.489930 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:46.489893    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0" path="/var/lib/kubelet/pods/ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0/volumes"
Jul 31 15:17:48.476520 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:48.476492    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"
Jul 31 15:17:55.183293 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:17:55.183255    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"]
Jul 31 15:18:10.201830 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.201786    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s" podUID="4f84fe81-97e0-4179-8613-60d959af937a" containerName="main" containerID="cri-o://717c66d126423d56c52d610d27ee85ad9c85e536b3469fd2c0456aa383d15c1e" gracePeriod=45
Jul 31 15:18:10.330653 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.330615    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s" podUID="4f84fe81-97e0-4179-8613-60d959af937a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 31 15:18:10.447762 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.447732    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:18:10.577260 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.577223    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f84fe81-97e0-4179-8613-60d959af937a-tls-certs\") pod \"4f84fe81-97e0-4179-8613-60d959af937a\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") "
Jul 31 15:18:10.577495 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.577318    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-dshm\") pod \"4f84fe81-97e0-4179-8613-60d959af937a\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") "
Jul 31 15:18:10.577495 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.577363    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-tmp-dir\") pod \"4f84fe81-97e0-4179-8613-60d959af937a\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") "
Jul 31 15:18:10.577495 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.577423    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-model-cache\") pod \"4f84fe81-97e0-4179-8613-60d959af937a\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") "
Jul 31 15:18:10.577495 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.577451    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-home\") pod \"4f84fe81-97e0-4179-8613-60d959af937a\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") "
Jul 31 15:18:10.577721 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.577498    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52rc8\" (UniqueName: \"kubernetes.io/projected/4f84fe81-97e0-4179-8613-60d959af937a-kube-api-access-52rc8\") pod \"4f84fe81-97e0-4179-8613-60d959af937a\" (UID: \"4f84fe81-97e0-4179-8613-60d959af937a\") "
Jul 31 15:18:10.578366 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.577911    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f84fe81-97e0-4179-8613-60d959af937a" (UID: "4f84fe81-97e0-4179-8613-60d959af937a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:18:10.578366 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.578223    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-home" (OuterVolumeSpecName: "home") pod "4f84fe81-97e0-4179-8613-60d959af937a" (UID: "4f84fe81-97e0-4179-8613-60d959af937a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:18:10.578366 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.578293    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f84fe81-97e0-4179-8613-60d959af937a" (UID: "4f84fe81-97e0-4179-8613-60d959af937a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:18:10.580401 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.580311    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-dshm" (OuterVolumeSpecName: "dshm") pod "4f84fe81-97e0-4179-8613-60d959af937a" (UID: "4f84fe81-97e0-4179-8613-60d959af937a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:18:10.580401 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.580318    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f84fe81-97e0-4179-8613-60d959af937a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f84fe81-97e0-4179-8613-60d959af937a" (UID: "4f84fe81-97e0-4179-8613-60d959af937a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:18:10.580590 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.580415    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f84fe81-97e0-4179-8613-60d959af937a-kube-api-access-52rc8" (OuterVolumeSpecName: "kube-api-access-52rc8") pod "4f84fe81-97e0-4179-8613-60d959af937a" (UID: "4f84fe81-97e0-4179-8613-60d959af937a"). InnerVolumeSpecName "kube-api-access-52rc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:18:10.677138 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.677105    2578 generic.go:356] "Generic (PLEG): container finished" podID="4f84fe81-97e0-4179-8613-60d959af937a" containerID="717c66d126423d56c52d610d27ee85ad9c85e536b3469fd2c0456aa383d15c1e" exitCode=0
Jul 31 15:18:10.677310 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.677153    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"
Jul 31 15:18:10.677310 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.677163    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s" event={"ID":"4f84fe81-97e0-4179-8613-60d959af937a","Type":"ContainerDied","Data":"717c66d126423d56c52d610d27ee85ad9c85e536b3469fd2c0456aa383d15c1e"}
Jul 31 15:18:10.677310 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.677207    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s" event={"ID":"4f84fe81-97e0-4179-8613-60d959af937a","Type":"ContainerDied","Data":"9cb63cb6513ed793dad5a0df056b236b049de482f77df0d874343b5df06f2d12"}
Jul 31 15:18:10.677310 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.677227    2578 scope.go:117] "RemoveContainer" containerID="717c66d126423d56c52d610d27ee85ad9c85e536b3469fd2c0456aa383d15c1e"
Jul 31 15:18:10.678033 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.678012    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:18:10.678033 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.678033    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:18:10.678177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.678043    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52rc8\" (UniqueName: \"kubernetes.io/projected/4f84fe81-97e0-4179-8613-60d959af937a-kube-api-access-52rc8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:18:10.678177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.678053    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f84fe81-97e0-4179-8613-60d959af937a-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:18:10.678177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.678063    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:18:10.678177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.678093    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f84fe81-97e0-4179-8613-60d959af937a-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:18:10.686637 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.686618    2578 scope.go:117] "RemoveContainer" containerID="717c66d126423d56c52d610d27ee85ad9c85e536b3469fd2c0456aa383d15c1e"
Jul 31 15:18:10.686901 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:18:10.686882    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"717c66d126423d56c52d610d27ee85ad9c85e536b3469fd2c0456aa383d15c1e\": container with ID starting with 717c66d126423d56c52d610d27ee85ad9c85e536b3469fd2c0456aa383d15c1e not found: ID does not exist" containerID="717c66d126423d56c52d610d27ee85ad9c85e536b3469fd2c0456aa383d15c1e"
Jul 31 15:18:10.686978 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.686916    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"717c66d126423d56c52d610d27ee85ad9c85e536b3469fd2c0456aa383d15c1e"} err="failed to get container status \"717c66d126423d56c52d610d27ee85ad9c85e536b3469fd2c0456aa383d15c1e\": rpc error: code = NotFound desc = could not find container \"717c66d126423d56c52d610d27ee85ad9c85e536b3469fd2c0456aa383d15c1e\": container with ID starting with 717c66d126423d56c52d610d27ee85ad9c85e536b3469fd2c0456aa383d15c1e not found: ID does not exist"
Jul 31 15:18:10.702445 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.702421    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"]
Jul 31 15:18:10.706325 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:10.706303    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-zbb9s"]
Jul 31 15:18:12.489406 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:12.489373    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f84fe81-97e0-4179-8613-60d959af937a" path="/var/lib/kubelet/pods/4f84fe81-97e0-4179-8613-60d959af937a/volumes"
Jul 31 15:18:38.045842 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:38.045810    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"]
Jul 31 15:18:38.475812 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:38.475774    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm" podUID="122713a8-44de-40f7-b8ed-89799c539df3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 15:18:48.476029 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:48.475987    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm" podUID="122713a8-44de-40f7-b8ed-89799c539df3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 15:18:53.064170 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:53.064120    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm" podUID="122713a8-44de-40f7-b8ed-89799c539df3" containerName="main" containerID="cri-o://08dbaad1d65f64900710385b2ef89188e8f771e052941f60988244f441401b09" gracePeriod=45
Jul 31 15:18:53.307051 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:53.307024    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"
Jul 31 15:18:53.420386 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:53.420272    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk6bx\" (UniqueName: \"kubernetes.io/projected/122713a8-44de-40f7-b8ed-89799c539df3-kube-api-access-fk6bx\") pod \"122713a8-44de-40f7-b8ed-89799c539df3\" (UID: \"122713a8-44de-40f7-b8ed-89799c539df3\") "
Jul 31 15:18:53.420562 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:53.420397    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/122713a8-44de-40f7-b8ed-89799c539df3-tls-certs\") pod \"122713a8-44de-40f7-b8ed-89799c539df3\" (UID: \"122713a8-44de-40f7-b8ed-89799c539df3\") "
Jul 31 15:18:53.422568 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:53.422534    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122713a8-44de-40f7-b8ed-89799c539df3-kube-api-access-fk6bx" (OuterVolumeSpecName: "kube-api-access-fk6bx") pod "122713a8-44de-40f7-b8ed-89799c539df3" (UID: "122713a8-44de-40f7-b8ed-89799c539df3"). InnerVolumeSpecName "kube-api-access-fk6bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:18:53.422568 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:53.422556    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/122713a8-44de-40f7-b8ed-89799c539df3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "122713a8-44de-40f7-b8ed-89799c539df3" (UID: "122713a8-44de-40f7-b8ed-89799c539df3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:18:53.521456 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:53.521415    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/122713a8-44de-40f7-b8ed-89799c539df3-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:18:53.521456 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:53.521449    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk6bx\" (UniqueName: \"kubernetes.io/projected/122713a8-44de-40f7-b8ed-89799c539df3-kube-api-access-fk6bx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:18:53.841638 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:53.841599    2578 generic.go:356] "Generic (PLEG): container finished" podID="122713a8-44de-40f7-b8ed-89799c539df3" containerID="08dbaad1d65f64900710385b2ef89188e8f771e052941f60988244f441401b09" exitCode=0
Jul 31 15:18:53.841899 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:53.841646    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"
Jul 31 15:18:53.841899 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:53.841655    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm" event={"ID":"122713a8-44de-40f7-b8ed-89799c539df3","Type":"ContainerDied","Data":"08dbaad1d65f64900710385b2ef89188e8f771e052941f60988244f441401b09"}
Jul 31 15:18:53.841899 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:53.841705    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm" event={"ID":"122713a8-44de-40f7-b8ed-89799c539df3","Type":"ContainerDied","Data":"2e3d01d2da66239a2678a901d33a402dc77d900c8a41145fc390c4e5010a5d8e"}
Jul 31 15:18:53.841899 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:53.841730    2578 scope.go:117] "RemoveContainer" containerID="08dbaad1d65f64900710385b2ef89188e8f771e052941f60988244f441401b09"
Jul 31 15:18:53.850947 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:53.850928    2578 scope.go:117] "RemoveContainer" containerID="08dbaad1d65f64900710385b2ef89188e8f771e052941f60988244f441401b09"
Jul 31 15:18:53.851214 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:18:53.851189    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08dbaad1d65f64900710385b2ef89188e8f771e052941f60988244f441401b09\": container with ID starting with 08dbaad1d65f64900710385b2ef89188e8f771e052941f60988244f441401b09 not found: ID does not exist" containerID="08dbaad1d65f64900710385b2ef89188e8f771e052941f60988244f441401b09"
Jul 31 15:18:53.851262 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:53.851222    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08dbaad1d65f64900710385b2ef89188e8f771e052941f60988244f441401b09"} err="failed to get container status \"08dbaad1d65f64900710385b2ef89188e8f771e052941f60988244f441401b09\": rpc error: code = NotFound desc = could not find container \"08dbaad1d65f64900710385b2ef89188e8f771e052941f60988244f441401b09\": container with ID starting with 08dbaad1d65f64900710385b2ef89188e8f771e052941f60988244f441401b09 not found: ID does not exist"
Jul 31 15:18:53.864041 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:53.864012    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"]
Jul 31 15:18:53.867406 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:53.867384    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schewt7sm"]
Jul 31 15:18:54.490292 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:54.490258    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="122713a8-44de-40f7-b8ed-89799c539df3" path="/var/lib/kubelet/pods/122713a8-44de-40f7-b8ed-89799c539df3/volumes"
Jul 31 15:18:59.303521 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.303485    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"]
Jul 31 15:18:59.303886 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.303749    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f84fe81-97e0-4179-8613-60d959af937a" containerName="main"
Jul 31 15:18:59.303886 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.303760    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f84fe81-97e0-4179-8613-60d959af937a" containerName="main"
Jul 31 15:18:59.303886 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.303768    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0" containerName="main"
Jul 31 15:18:59.303886 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.303773    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0" containerName="main"
Jul 31 15:18:59.303886 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.303785    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="122713a8-44de-40f7-b8ed-89799c539df3" containerName="main"
Jul 31 15:18:59.303886 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.303791    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="122713a8-44de-40f7-b8ed-89799c539df3" containerName="main"
Jul 31 15:18:59.303886 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.303836    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f84fe81-97e0-4179-8613-60d959af937a" containerName="main"
Jul 31 15:18:59.303886 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.303842    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca863ff7-b2b1-4c03-a3b8-8a9bb05396a0" containerName="main"
Jul 31 15:18:59.303886 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.303850    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="122713a8-44de-40f7-b8ed-89799c539df3" containerName="main"
Jul 31 15:18:59.306234 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.306215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.309155 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.309129    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-qptkq\""
Jul 31 15:18:59.309469 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.309449    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 31 15:18:59.310607 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.310588    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 31 15:18:59.310699 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.310589    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 31 15:18:59.323525 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.323498    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"]
Jul 31 15:18:59.472765 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.472706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.472765 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.472765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.472993 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.472837    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b023f2b5-304c-43d6-86b7-1f5e44359bb5-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.472993 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.472865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5n2l\" (UniqueName: \"kubernetes.io/projected/b023f2b5-304c-43d6-86b7-1f5e44359bb5-kube-api-access-j5n2l\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.472993 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.472904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.472993 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.472927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.573566 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.573458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b023f2b5-304c-43d6-86b7-1f5e44359bb5-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.573566 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.573525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5n2l\" (UniqueName: \"kubernetes.io/projected/b023f2b5-304c-43d6-86b7-1f5e44359bb5-kube-api-access-j5n2l\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.573566 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.573559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.573566 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.573577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.573984 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.573942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.574123 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.574001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.574123 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.574022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.574123 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.574081    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.574365 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.574331    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.576155 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.576132    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.576370 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.576339    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b023f2b5-304c-43d6-86b7-1f5e44359bb5-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.584170 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.584149    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5n2l\" (UniqueName: \"kubernetes.io/projected/b023f2b5-304c-43d6-86b7-1f5e44359bb5-kube-api-access-j5n2l\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.616202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.616157    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:18:59.750465 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.750436    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"]
Jul 31 15:18:59.752951 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:18:59.752926    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb023f2b5_304c_43d6_86b7_1f5e44359bb5.slice/crio-6eedf27e27ae1ea6b84027e6aa04da182975d929677f98d7f5b4a2450106a242 WatchSource:0}: Error finding container 6eedf27e27ae1ea6b84027e6aa04da182975d929677f98d7f5b4a2450106a242: Status 404 returned error can't find the container with id 6eedf27e27ae1ea6b84027e6aa04da182975d929677f98d7f5b4a2450106a242
Jul 31 15:18:59.866336 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.866245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx" event={"ID":"b023f2b5-304c-43d6-86b7-1f5e44359bb5","Type":"ContainerStarted","Data":"4f45f31e6942495f922f5701a69ba5bb9e16e67a94414fc4fdad14f49d3d9571"}
Jul 31 15:18:59.866336 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.866305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx" event={"ID":"b023f2b5-304c-43d6-86b7-1f5e44359bb5","Type":"ContainerStarted","Data":"6eedf27e27ae1ea6b84027e6aa04da182975d929677f98d7f5b4a2450106a242"}
Jul 31 15:18:59.887027 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:18:59.886978    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx" podStartSLOduration=0.886965047 podStartE2EDuration="886.965047ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:18:59.885901289 +0000 UTC m=+1837.895113159" watchObservedRunningTime="2026-07-31 15:18:59.886965047 +0000 UTC m=+1837.896176916"
Jul 31 15:19:00.617084 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:00.617050    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:19:09.616454 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:09.616401    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:19:09.628708 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:09.628677    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:19:09.918883 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:09.918800    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:19:36.895263 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:36.895220    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"]
Jul 31 15:19:36.903249 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:36.903220    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:36.905979 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:36.905947    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Jul 31 15:19:36.907100 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:36.907077    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Jul 31 15:19:36.907230 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:36.907077    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Jul 31 15:19:36.907230 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:36.907140    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-zww8c\""
Jul 31 15:19:36.907657 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:36.907638    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"]
Jul 31 15:19:36.978438 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:36.978407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkwj7\" (UniqueName: \"kubernetes.io/projected/9133718b-d57f-4369-b2f1-cfba0900d005-kube-api-access-bkwj7\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:36.978596 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:36.978446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:36.978596 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:36.978479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-model-cache\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:36.978596 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:36.978557    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-dshm\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:36.978705 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:36.978602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-home\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:36.978705 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:36.978627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9133718b-d57f-4369-b2f1-cfba0900d005-tls-certs\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:37.080028 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:37.079990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwj7\" (UniqueName: \"kubernetes.io/projected/9133718b-d57f-4369-b2f1-cfba0900d005-kube-api-access-bkwj7\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:37.080028 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:37.080036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:37.080275 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:37.080247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-model-cache\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:37.080327 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:37.080311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-dshm\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:37.080397 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:37.080376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-home\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:37.080449 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:37.080421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9133718b-d57f-4369-b2f1-cfba0900d005-tls-certs\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:37.080574 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:37.080555    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:37.080665 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:37.080643    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-model-cache\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:37.080772 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:37.080744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-home\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:37.082825 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:37.082806    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-dshm\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:37.083162 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:37.083146    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9133718b-d57f-4369-b2f1-cfba0900d005-tls-certs\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:37.088357 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:37.088328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkwj7\" (UniqueName: \"kubernetes.io/projected/9133718b-d57f-4369-b2f1-cfba0900d005-kube-api-access-bkwj7\") pod \"leave-v1-kserve-65684d4f5-7f4dt\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:37.215647 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:37.215542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:37.343952 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:37.343925    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"]
Jul 31 15:19:37.346302 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:19:37.346274    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9133718b_d57f_4369_b2f1_cfba0900d005.slice/crio-7d7abe6e51637fe7854b5149d969b4f7877aa3accb9235beb97fe7ab9e69ebcf WatchSource:0}: Error finding container 7d7abe6e51637fe7854b5149d969b4f7877aa3accb9235beb97fe7ab9e69ebcf: Status 404 returned error can't find the container with id 7d7abe6e51637fe7854b5149d969b4f7877aa3accb9235beb97fe7ab9e69ebcf
Jul 31 15:19:38.011192 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:38.011151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt" event={"ID":"9133718b-d57f-4369-b2f1-cfba0900d005","Type":"ContainerStarted","Data":"f2b0108a6156faddc8509381ee133a0480559faf8dbc23ed7c88fe5a9a09ebba"}
Jul 31 15:19:38.011682 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:38.011205    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt" event={"ID":"9133718b-d57f-4369-b2f1-cfba0900d005","Type":"ContainerStarted","Data":"7d7abe6e51637fe7854b5149d969b4f7877aa3accb9235beb97fe7ab9e69ebcf"}
Jul 31 15:19:38.030181 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:38.030131    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt" podStartSLOduration=2.030116299 podStartE2EDuration="2.030116299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:19:38.029794226 +0000 UTC m=+1876.039006108" watchObservedRunningTime="2026-07-31 15:19:38.030116299 +0000 UTC m=+1876.039328169"
Jul 31 15:19:38.216290 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:38.216256    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:47.216317 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:47.216231    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:47.228946 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:47.228921    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:48.060157 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:48.060122    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:19:59.539461 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:59.539426    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"]
Jul 31 15:19:59.542547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:59.542526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"
Jul 31 15:19:59.545429 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:59.545405    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 31 15:19:59.545665 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:59.545646    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-fgpjc\""
Jul 31 15:19:59.558260 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:59.558232    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"]
Jul 31 15:19:59.676165 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:59.676115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-282h6\" (UniqueName: \"kubernetes.io/projected/041a1281-65b0-4545-8083-7595b1885e4b-kube-api-access-282h6\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487\" (UID: \"041a1281-65b0-4545-8083-7595b1885e4b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"
Jul 31 15:19:59.676412 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:59.676248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/041a1281-65b0-4545-8083-7595b1885e4b-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487\" (UID: \"041a1281-65b0-4545-8083-7595b1885e4b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"
Jul 31 15:19:59.777604 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:59.777561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-282h6\" (UniqueName: \"kubernetes.io/projected/041a1281-65b0-4545-8083-7595b1885e4b-kube-api-access-282h6\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487\" (UID: \"041a1281-65b0-4545-8083-7595b1885e4b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"
Jul 31 15:19:59.777787 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:59.777663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/041a1281-65b0-4545-8083-7595b1885e4b-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487\" (UID: \"041a1281-65b0-4545-8083-7595b1885e4b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"
Jul 31 15:19:59.780444 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:59.780417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/041a1281-65b0-4545-8083-7595b1885e4b-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487\" (UID: \"041a1281-65b0-4545-8083-7595b1885e4b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"
Jul 31 15:19:59.792983 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:59.792920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-282h6\" (UniqueName: \"kubernetes.io/projected/041a1281-65b0-4545-8083-7595b1885e4b-kube-api-access-282h6\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487\" (UID: \"041a1281-65b0-4545-8083-7595b1885e4b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"
Jul 31 15:19:59.855723 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:59.855696    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"
Jul 31 15:19:59.992360 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:19:59.992321    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"]
Jul 31 15:19:59.994006 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:19:59.993974    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041a1281_65b0_4545_8083_7595b1885e4b.slice/crio-b1d0fb66c9f5e0cbf6a49e30a13975fdd393c1d95b960f2c5336a9e92ecab122 WatchSource:0}: Error finding container b1d0fb66c9f5e0cbf6a49e30a13975fdd393c1d95b960f2c5336a9e92ecab122: Status 404 returned error can't find the container with id b1d0fb66c9f5e0cbf6a49e30a13975fdd393c1d95b960f2c5336a9e92ecab122
Jul 31 15:20:00.101716 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:00.101680    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487" event={"ID":"041a1281-65b0-4545-8083-7595b1885e4b","Type":"ContainerStarted","Data":"859354e9098fa5169e3e6dc3270a031b32361c7e8a7d6e94f1744c757657505c"}
Jul 31 15:20:00.101914 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:00.101744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487" event={"ID":"041a1281-65b0-4545-8083-7595b1885e4b","Type":"ContainerStarted","Data":"b1d0fb66c9f5e0cbf6a49e30a13975fdd393c1d95b960f2c5336a9e92ecab122"}
Jul 31 15:20:00.101914 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:00.101874    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"
Jul 31 15:20:00.130627 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:00.130578    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487" podStartSLOduration=1.130561534 podStartE2EDuration="1.130561534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:20:00.127728031 +0000 UTC m=+1898.136939925" watchObservedRunningTime="2026-07-31 15:20:00.130561534 +0000 UTC m=+1898.139773403"
Jul 31 15:20:31.107670 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:31.107640    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"
Jul 31 15:20:34.448531 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:34.448492    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"]
Jul 31 15:20:49.467221 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.467178    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt" podUID="9133718b-d57f-4369-b2f1-cfba0900d005" containerName="main" containerID="cri-o://f2b0108a6156faddc8509381ee133a0480559faf8dbc23ed7c88fe5a9a09ebba" gracePeriod=45
Jul 31 15:20:49.712918 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.712890    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:20:49.783937 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.783907    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9133718b-d57f-4369-b2f1-cfba0900d005-tls-certs\") pod \"9133718b-d57f-4369-b2f1-cfba0900d005\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") "
Jul 31 15:20:49.784117 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.783968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkwj7\" (UniqueName: \"kubernetes.io/projected/9133718b-d57f-4369-b2f1-cfba0900d005-kube-api-access-bkwj7\") pod \"9133718b-d57f-4369-b2f1-cfba0900d005\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") "
Jul 31 15:20:49.784117 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.784019    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-model-cache\") pod \"9133718b-d57f-4369-b2f1-cfba0900d005\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") "
Jul 31 15:20:49.784117 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.784058    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-dshm\") pod \"9133718b-d57f-4369-b2f1-cfba0900d005\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") "
Jul 31 15:20:49.784276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.784255    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-home\") pod \"9133718b-d57f-4369-b2f1-cfba0900d005\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") "
Jul 31 15:20:49.784419 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.784315    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-tmp-dir\") pod \"9133718b-d57f-4369-b2f1-cfba0900d005\" (UID: \"9133718b-d57f-4369-b2f1-cfba0900d005\") "
Jul 31 15:20:49.784493 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.784310    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-model-cache" (OuterVolumeSpecName: "model-cache") pod "9133718b-d57f-4369-b2f1-cfba0900d005" (UID: "9133718b-d57f-4369-b2f1-cfba0900d005"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:20:49.784661 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.784612    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9133718b-d57f-4369-b2f1-cfba0900d005" (UID: "9133718b-d57f-4369-b2f1-cfba0900d005"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:20:49.784661 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.784625    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:20:49.784661 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.784637    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-home" (OuterVolumeSpecName: "home") pod "9133718b-d57f-4369-b2f1-cfba0900d005" (UID: "9133718b-d57f-4369-b2f1-cfba0900d005"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:20:49.786180 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.786160    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9133718b-d57f-4369-b2f1-cfba0900d005-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9133718b-d57f-4369-b2f1-cfba0900d005" (UID: "9133718b-d57f-4369-b2f1-cfba0900d005"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:20:49.786474 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.786459    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-dshm" (OuterVolumeSpecName: "dshm") pod "9133718b-d57f-4369-b2f1-cfba0900d005" (UID: "9133718b-d57f-4369-b2f1-cfba0900d005"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:20:49.786534 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.786491    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9133718b-d57f-4369-b2f1-cfba0900d005-kube-api-access-bkwj7" (OuterVolumeSpecName: "kube-api-access-bkwj7") pod "9133718b-d57f-4369-b2f1-cfba0900d005" (UID: "9133718b-d57f-4369-b2f1-cfba0900d005"). InnerVolumeSpecName "kube-api-access-bkwj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:20:49.885578 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.885547    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:20:49.885578 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.885575    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:20:49.885578 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.885585    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9133718b-d57f-4369-b2f1-cfba0900d005-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:20:49.885809 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.885593    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9133718b-d57f-4369-b2f1-cfba0900d005-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:20:49.885809 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:49.885602    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkwj7\" (UniqueName: \"kubernetes.io/projected/9133718b-d57f-4369-b2f1-cfba0900d005-kube-api-access-bkwj7\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:20:50.300476 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:50.300440    2578 generic.go:356] "Generic (PLEG): container finished" podID="9133718b-d57f-4369-b2f1-cfba0900d005" containerID="f2b0108a6156faddc8509381ee133a0480559faf8dbc23ed7c88fe5a9a09ebba" exitCode=0
Jul 31 15:20:50.300656 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:50.300482    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"
Jul 31 15:20:50.300656 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:50.300507    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt" event={"ID":"9133718b-d57f-4369-b2f1-cfba0900d005","Type":"ContainerDied","Data":"f2b0108a6156faddc8509381ee133a0480559faf8dbc23ed7c88fe5a9a09ebba"}
Jul 31 15:20:50.300656 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:50.300551    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt" event={"ID":"9133718b-d57f-4369-b2f1-cfba0900d005","Type":"ContainerDied","Data":"7d7abe6e51637fe7854b5149d969b4f7877aa3accb9235beb97fe7ab9e69ebcf"}
Jul 31 15:20:50.300656 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:50.300575    2578 scope.go:117] "RemoveContainer" containerID="f2b0108a6156faddc8509381ee133a0480559faf8dbc23ed7c88fe5a9a09ebba"
Jul 31 15:20:50.308910 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:50.308889    2578 scope.go:117] "RemoveContainer" containerID="f2b0108a6156faddc8509381ee133a0480559faf8dbc23ed7c88fe5a9a09ebba"
Jul 31 15:20:50.309178 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:20:50.309156    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2b0108a6156faddc8509381ee133a0480559faf8dbc23ed7c88fe5a9a09ebba\": container with ID starting with f2b0108a6156faddc8509381ee133a0480559faf8dbc23ed7c88fe5a9a09ebba not found: ID does not exist" containerID="f2b0108a6156faddc8509381ee133a0480559faf8dbc23ed7c88fe5a9a09ebba"
Jul 31 15:20:50.309224 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:50.309186    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b0108a6156faddc8509381ee133a0480559faf8dbc23ed7c88fe5a9a09ebba"} err="failed to get container status \"f2b0108a6156faddc8509381ee133a0480559faf8dbc23ed7c88fe5a9a09ebba\": rpc error: code = NotFound desc = could not find container \"f2b0108a6156faddc8509381ee133a0480559faf8dbc23ed7c88fe5a9a09ebba\": container with ID starting with f2b0108a6156faddc8509381ee133a0480559faf8dbc23ed7c88fe5a9a09ebba not found: ID does not exist"
Jul 31 15:20:50.326391 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:50.323308    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"]
Jul 31 15:20:50.327436 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:50.327408    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-7f4dt"]
Jul 31 15:20:50.489469 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:20:50.489425    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9133718b-d57f-4369-b2f1-cfba0900d005" path="/var/lib/kubelet/pods/9133718b-d57f-4369-b2f1-cfba0900d005/volumes"
Jul 31 15:21:08.886158 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:08.886120    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"]
Jul 31 15:21:08.886623 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:08.886604    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9133718b-d57f-4369-b2f1-cfba0900d005" containerName="main"
Jul 31 15:21:08.886678 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:08.886627    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9133718b-d57f-4369-b2f1-cfba0900d005" containerName="main"
Jul 31 15:21:08.886725 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:08.886714    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9133718b-d57f-4369-b2f1-cfba0900d005" containerName="main"
Jul 31 15:21:08.889883 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:08.889866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:08.892883 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:08.892865    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Jul 31 15:21:08.894129 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:08.894112    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Jul 31 15:21:08.894239 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:08.894154    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-h2rbf\""
Jul 31 15:21:08.894416 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:08.894401    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Jul 31 15:21:08.898981 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:08.898960    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"]
Jul 31 15:21:09.046021 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.045980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-tls-certs\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.046021 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.046024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-home\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.046237 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.046046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-dshm\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.046237 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.046098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.046237 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.046117    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxv7f\" (UniqueName: \"kubernetes.io/projected/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-kube-api-access-qxv7f\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.046237 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.046210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-model-cache\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.147408 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.147301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-tls-certs\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.147408 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.147342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-home\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.147408 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.147398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-dshm\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.147665 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.147425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.147665 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.147447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxv7f\" (UniqueName: \"kubernetes.io/projected/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-kube-api-access-qxv7f\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.147665 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.147507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-model-cache\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.147916 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.147892    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-model-cache\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.147993 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.147919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-home\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.147993 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.147941    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.149886 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.149861    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-dshm\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.150025 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.150002    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-tls-certs\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.156663 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.156642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxv7f\" (UniqueName: \"kubernetes.io/projected/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-kube-api-access-qxv7f\") pod \"tri-v3-kserve-798c88c66c-nwldl\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.201232 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.201195    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:09.331703 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.331677    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"]
Jul 31 15:21:09.334623 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:21:09.334583    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1aeba4_9b76_4a2c_9d6e_3d21a7134e22.slice/crio-501d97e83f836d9580bbd02622edfd50232566740283edb24f16b7a784a7d331 WatchSource:0}: Error finding container 501d97e83f836d9580bbd02622edfd50232566740283edb24f16b7a784a7d331: Status 404 returned error can't find the container with id 501d97e83f836d9580bbd02622edfd50232566740283edb24f16b7a784a7d331
Jul 31 15:21:09.375619 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:09.375576    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl" event={"ID":"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22","Type":"ContainerStarted","Data":"501d97e83f836d9580bbd02622edfd50232566740283edb24f16b7a784a7d331"}
Jul 31 15:21:10.381523 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:10.381485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl" event={"ID":"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22","Type":"ContainerStarted","Data":"546ac6deaec8f69eff36d0f31ffa4a0674125a26b61a792709008f35e529b9b1"}
Jul 31 15:21:10.400603 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:10.400554    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl" podStartSLOduration=2.4005401920000002 podStartE2EDuration="2.400540192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:21:10.3995979 +0000 UTC m=+1968.408809791" watchObservedRunningTime="2026-07-31 15:21:10.400540192 +0000 UTC m=+1968.409752060"
Jul 31 15:21:11.201779 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:11.201741    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:19.201695 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:19.201614    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:19.214099 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:19.214072    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:19.428961 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:19.428933    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:21:23.915211 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:23.915174    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"]
Jul 31 15:21:31.107514 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:31.107468    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487" podUID="041a1281-65b0-4545-8083-7595b1885e4b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 15:21:38.935898 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:38.935835    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487" podUID="041a1281-65b0-4545-8083-7595b1885e4b" containerName="main" containerID="cri-o://859354e9098fa5169e3e6dc3270a031b32361c7e8a7d6e94f1744c757657505c" gracePeriod=45
Jul 31 15:21:39.171118 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:39.171095    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"
Jul 31 15:21:39.184119 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:39.184094    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-282h6\" (UniqueName: \"kubernetes.io/projected/041a1281-65b0-4545-8083-7595b1885e4b-kube-api-access-282h6\") pod \"041a1281-65b0-4545-8083-7595b1885e4b\" (UID: \"041a1281-65b0-4545-8083-7595b1885e4b\") "
Jul 31 15:21:39.184270 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:39.184130    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/041a1281-65b0-4545-8083-7595b1885e4b-tls-certs\") pod \"041a1281-65b0-4545-8083-7595b1885e4b\" (UID: \"041a1281-65b0-4545-8083-7595b1885e4b\") "
Jul 31 15:21:39.186291 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:39.186231    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041a1281-65b0-4545-8083-7595b1885e4b-kube-api-access-282h6" (OuterVolumeSpecName: "kube-api-access-282h6") pod "041a1281-65b0-4545-8083-7595b1885e4b" (UID: "041a1281-65b0-4545-8083-7595b1885e4b"). InnerVolumeSpecName "kube-api-access-282h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:21:39.186412 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:39.186305    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/041a1281-65b0-4545-8083-7595b1885e4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "041a1281-65b0-4545-8083-7595b1885e4b" (UID: "041a1281-65b0-4545-8083-7595b1885e4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:21:39.284788 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:39.284744    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-282h6\" (UniqueName: \"kubernetes.io/projected/041a1281-65b0-4545-8083-7595b1885e4b-kube-api-access-282h6\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:21:39.284788 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:39.284789    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/041a1281-65b0-4545-8083-7595b1885e4b-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:21:39.498416 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:39.498383    2578 generic.go:356] "Generic (PLEG): container finished" podID="041a1281-65b0-4545-8083-7595b1885e4b" containerID="859354e9098fa5169e3e6dc3270a031b32361c7e8a7d6e94f1744c757657505c" exitCode=0
Jul 31 15:21:39.498584 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:39.498478    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"
Jul 31 15:21:39.498584 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:39.498475    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487" event={"ID":"041a1281-65b0-4545-8083-7595b1885e4b","Type":"ContainerDied","Data":"859354e9098fa5169e3e6dc3270a031b32361c7e8a7d6e94f1744c757657505c"}
Jul 31 15:21:39.498665 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:39.498606    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487" event={"ID":"041a1281-65b0-4545-8083-7595b1885e4b","Type":"ContainerDied","Data":"b1d0fb66c9f5e0cbf6a49e30a13975fdd393c1d95b960f2c5336a9e92ecab122"}
Jul 31 15:21:39.498665 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:39.498637    2578 scope.go:117] "RemoveContainer" containerID="859354e9098fa5169e3e6dc3270a031b32361c7e8a7d6e94f1744c757657505c"
Jul 31 15:21:39.507312 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:39.507289    2578 scope.go:117] "RemoveContainer" containerID="859354e9098fa5169e3e6dc3270a031b32361c7e8a7d6e94f1744c757657505c"
Jul 31 15:21:39.507562 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:21:39.507542    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"859354e9098fa5169e3e6dc3270a031b32361c7e8a7d6e94f1744c757657505c\": container with ID starting with 859354e9098fa5169e3e6dc3270a031b32361c7e8a7d6e94f1744c757657505c not found: ID does not exist" containerID="859354e9098fa5169e3e6dc3270a031b32361c7e8a7d6e94f1744c757657505c"
Jul 31 15:21:39.507647 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:39.507569    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859354e9098fa5169e3e6dc3270a031b32361c7e8a7d6e94f1744c757657505c"} err="failed to get container status \"859354e9098fa5169e3e6dc3270a031b32361c7e8a7d6e94f1744c757657505c\": rpc error: code = NotFound desc = could not find container \"859354e9098fa5169e3e6dc3270a031b32361c7e8a7d6e94f1744c757657505c\": container with ID starting with 859354e9098fa5169e3e6dc3270a031b32361c7e8a7d6e94f1744c757657505c not found: ID does not exist"
Jul 31 15:21:39.521783 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:39.521754    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"]
Jul 31 15:21:39.527676 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:39.527654    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schebw487"]
Jul 31 15:21:40.489029 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:40.488993    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="041a1281-65b0-4545-8083-7595b1885e4b" path="/var/lib/kubelet/pods/041a1281-65b0-4545-8083-7595b1885e4b/volumes"
Jul 31 15:21:42.877576 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:42.877540    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"]
Jul 31 15:21:42.886171 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:21:42.886136    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:21:42.886171 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:21:42.886171 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:21:42.886171 ip-10-0-133-32 kubenswrapper[2578]:  > containerID="4f45f31e6942495f922f5701a69ba5bb9e16e67a94414fc4fdad14f49d3d9571" cmd=["/bin/sleep","15"]
Jul 31 15:21:42.886505 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:42.886193    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx" message="" err=<
Jul 31 15:21:42.886505 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:21:42.886505 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:21:42.886505 ip-10-0-133-32 kubenswrapper[2578]:  >
Jul 31 15:21:42.886505 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:21:42.886211    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:21:42.886505 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:21:42.886505 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:21:42.886505 ip-10-0-133-32 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx" podUID="b023f2b5-304c-43d6-86b7-1f5e44359bb5" containerName="main" containerID="cri-o://4f45f31e6942495f922f5701a69ba5bb9e16e67a94414fc4fdad14f49d3d9571"
Jul 31 15:21:42.886505 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:42.886266    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx" podUID="b023f2b5-304c-43d6-86b7-1f5e44359bb5" containerName="main" containerID="cri-o://4f45f31e6942495f922f5701a69ba5bb9e16e67a94414fc4fdad14f49d3d9571" gracePeriod=60
Jul 31 15:21:42.907985 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:42.907954    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx" podUID="b023f2b5-304c-43d6-86b7-1f5e44359bb5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 31 15:21:43.225936 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.225910    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:21:43.316091 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.316049    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-tmp-dir\") pod \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") "
Jul 31 15:21:43.316276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.316108    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-dshm\") pod \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") "
Jul 31 15:21:43.316276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.316151    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5n2l\" (UniqueName: \"kubernetes.io/projected/b023f2b5-304c-43d6-86b7-1f5e44359bb5-kube-api-access-j5n2l\") pod \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") "
Jul 31 15:21:43.316276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.316190    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-model-cache\") pod \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") "
Jul 31 15:21:43.316276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.316230    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-home\") pod \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") "
Jul 31 15:21:43.316276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.316263    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b023f2b5-304c-43d6-86b7-1f5e44359bb5-tls-certs\") pod \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\" (UID: \"b023f2b5-304c-43d6-86b7-1f5e44359bb5\") "
Jul 31 15:21:43.316583 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.316398    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b023f2b5-304c-43d6-86b7-1f5e44359bb5" (UID: "b023f2b5-304c-43d6-86b7-1f5e44359bb5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:21:43.316583 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.316509    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:21:43.316583 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.316553    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-home" (OuterVolumeSpecName: "home") pod "b023f2b5-304c-43d6-86b7-1f5e44359bb5" (UID: "b023f2b5-304c-43d6-86b7-1f5e44359bb5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:21:43.316583 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.316564    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-model-cache" (OuterVolumeSpecName: "model-cache") pod "b023f2b5-304c-43d6-86b7-1f5e44359bb5" (UID: "b023f2b5-304c-43d6-86b7-1f5e44359bb5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:21:43.318278 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.318256    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-dshm" (OuterVolumeSpecName: "dshm") pod "b023f2b5-304c-43d6-86b7-1f5e44359bb5" (UID: "b023f2b5-304c-43d6-86b7-1f5e44359bb5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:21:43.318394 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.318306    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b023f2b5-304c-43d6-86b7-1f5e44359bb5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b023f2b5-304c-43d6-86b7-1f5e44359bb5" (UID: "b023f2b5-304c-43d6-86b7-1f5e44359bb5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:21:43.318394 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.318317    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b023f2b5-304c-43d6-86b7-1f5e44359bb5-kube-api-access-j5n2l" (OuterVolumeSpecName: "kube-api-access-j5n2l") pod "b023f2b5-304c-43d6-86b7-1f5e44359bb5" (UID: "b023f2b5-304c-43d6-86b7-1f5e44359bb5"). InnerVolumeSpecName "kube-api-access-j5n2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:21:43.417004 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.416918    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:21:43.417004 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.416945    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b023f2b5-304c-43d6-86b7-1f5e44359bb5-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:21:43.417004 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.416955    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:21:43.417004 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.416964    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5n2l\" (UniqueName: \"kubernetes.io/projected/b023f2b5-304c-43d6-86b7-1f5e44359bb5-kube-api-access-j5n2l\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:21:43.417004 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.416973    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b023f2b5-304c-43d6-86b7-1f5e44359bb5-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:21:43.514739 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.514705    2578 generic.go:356] "Generic (PLEG): container finished" podID="b023f2b5-304c-43d6-86b7-1f5e44359bb5" containerID="4f45f31e6942495f922f5701a69ba5bb9e16e67a94414fc4fdad14f49d3d9571" exitCode=0
Jul 31 15:21:43.514920 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.514749    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"
Jul 31 15:21:43.514920 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.514754    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx" event={"ID":"b023f2b5-304c-43d6-86b7-1f5e44359bb5","Type":"ContainerDied","Data":"4f45f31e6942495f922f5701a69ba5bb9e16e67a94414fc4fdad14f49d3d9571"}
Jul 31 15:21:43.514920 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.514791    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx" event={"ID":"b023f2b5-304c-43d6-86b7-1f5e44359bb5","Type":"ContainerDied","Data":"6eedf27e27ae1ea6b84027e6aa04da182975d929677f98d7f5b4a2450106a242"}
Jul 31 15:21:43.514920 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.514813    2578 scope.go:117] "RemoveContainer" containerID="4f45f31e6942495f922f5701a69ba5bb9e16e67a94414fc4fdad14f49d3d9571"
Jul 31 15:21:43.523890 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.523873    2578 scope.go:117] "RemoveContainer" containerID="4f45f31e6942495f922f5701a69ba5bb9e16e67a94414fc4fdad14f49d3d9571"
Jul 31 15:21:43.524131 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:21:43.524112    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f45f31e6942495f922f5701a69ba5bb9e16e67a94414fc4fdad14f49d3d9571\": container with ID starting with 4f45f31e6942495f922f5701a69ba5bb9e16e67a94414fc4fdad14f49d3d9571 not found: ID does not exist" containerID="4f45f31e6942495f922f5701a69ba5bb9e16e67a94414fc4fdad14f49d3d9571"
Jul 31 15:21:43.524182 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.524140    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f45f31e6942495f922f5701a69ba5bb9e16e67a94414fc4fdad14f49d3d9571"} err="failed to get container status \"4f45f31e6942495f922f5701a69ba5bb9e16e67a94414fc4fdad14f49d3d9571\": rpc error: code = NotFound desc = could not find container \"4f45f31e6942495f922f5701a69ba5bb9e16e67a94414fc4fdad14f49d3d9571\": container with ID starting with 4f45f31e6942495f922f5701a69ba5bb9e16e67a94414fc4fdad14f49d3d9571 not found: ID does not exist"
Jul 31 15:21:43.564109 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.564072    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"]
Jul 31 15:21:43.567739 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:43.567714    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-wdpwx"]
Jul 31 15:21:44.492539 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:21:44.490929    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b023f2b5-304c-43d6-86b7-1f5e44359bb5" path="/var/lib/kubelet/pods/b023f2b5-304c-43d6-86b7-1f5e44359bb5/volumes"
Jul 31 15:22:06.735745 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.735708    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"]
Jul 31 15:22:06.736118 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.735979    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="041a1281-65b0-4545-8083-7595b1885e4b" containerName="main"
Jul 31 15:22:06.736118 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.735990    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="041a1281-65b0-4545-8083-7595b1885e4b" containerName="main"
Jul 31 15:22:06.736118 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.736000    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b023f2b5-304c-43d6-86b7-1f5e44359bb5" containerName="main"
Jul 31 15:22:06.736118 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.736006    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b023f2b5-304c-43d6-86b7-1f5e44359bb5" containerName="main"
Jul 31 15:22:06.736118 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.736054    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b023f2b5-304c-43d6-86b7-1f5e44359bb5" containerName="main"
Jul 31 15:22:06.736118 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.736070    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="041a1281-65b0-4545-8083-7595b1885e4b" containerName="main"
Jul 31 15:22:06.738894 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.738876    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.743151 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.743120    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-zrs6n\""
Jul 31 15:22:06.743151 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.743140    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 31 15:22:06.743310 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.743121    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 31 15:22:06.743310 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.743136    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 31 15:22:06.750390 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.750367    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"]
Jul 31 15:22:06.802382 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.802329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.802615 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.802450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.802615 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.802483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.802615 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.802508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.802615 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.802543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27d6e8b1-6c19-4882-95bc-701f98b87f59-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.802615 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.802560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8mxq\" (UniqueName: \"kubernetes.io/projected/27d6e8b1-6c19-4882-95bc-701f98b87f59-kube-api-access-q8mxq\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.903315 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.903273    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.903315 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.903316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.903581 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.903338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.903581 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.903405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27d6e8b1-6c19-4882-95bc-701f98b87f59-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.903581 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.903431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8mxq\" (UniqueName: \"kubernetes.io/projected/27d6e8b1-6c19-4882-95bc-701f98b87f59-kube-api-access-q8mxq\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.903581 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.903466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.903785 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.903762    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.903830 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.903802    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.903904 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.903885    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.905643 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.905615    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.906025 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.906003    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27d6e8b1-6c19-4882-95bc-701f98b87f59-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:06.914337 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:06.914309    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8mxq\" (UniqueName: \"kubernetes.io/projected/27d6e8b1-6c19-4882-95bc-701f98b87f59-kube-api-access-q8mxq\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:07.048964 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:07.048871    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:07.185217 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:07.185188    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"]
Jul 31 15:22:07.187746 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:22:07.187720    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d6e8b1_6c19_4882_95bc_701f98b87f59.slice/crio-99b50e3a0e1d23bfbf1caa3694e06604b4d7dd14e31f50228f0e2d8ca95afc30 WatchSource:0}: Error finding container 99b50e3a0e1d23bfbf1caa3694e06604b4d7dd14e31f50228f0e2d8ca95afc30: Status 404 returned error can't find the container with id 99b50e3a0e1d23bfbf1caa3694e06604b4d7dd14e31f50228f0e2d8ca95afc30
Jul 31 15:22:07.605524 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:07.605490    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9" event={"ID":"27d6e8b1-6c19-4882-95bc-701f98b87f59","Type":"ContainerStarted","Data":"eb2de1b37d29b23a67fdafc847daaa9811dbe69daf41f26d176a2d5f250ea0af"}
Jul 31 15:22:07.605713 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:07.605540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9" event={"ID":"27d6e8b1-6c19-4882-95bc-701f98b87f59","Type":"ContainerStarted","Data":"99b50e3a0e1d23bfbf1caa3694e06604b4d7dd14e31f50228f0e2d8ca95afc30"}
Jul 31 15:22:07.626439 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:07.626390    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9" podStartSLOduration=1.6263763340000001 podStartE2EDuration="1.626376334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:22:07.625589421 +0000 UTC m=+2025.634801290" watchObservedRunningTime="2026-07-31 15:22:07.626376334 +0000 UTC m=+2025.635588236"
Jul 31 15:22:08.049400 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:08.049342    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:17.049239 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:17.049197    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:17.061374 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:17.061326    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:17.659498 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:17.659470    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:22:19.465992 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:19.465950    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"]
Jul 31 15:22:34.484851 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.484799    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl" podUID="db1aeba4-9b76-4a2c-9d6e-3d21a7134e22" containerName="main" containerID="cri-o://546ac6deaec8f69eff36d0f31ffa4a0674125a26b61a792709008f35e529b9b1" gracePeriod=45
Jul 31 15:22:34.717541 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.717509    2578 generic.go:356] "Generic (PLEG): container finished" podID="db1aeba4-9b76-4a2c-9d6e-3d21a7134e22" containerID="546ac6deaec8f69eff36d0f31ffa4a0674125a26b61a792709008f35e529b9b1" exitCode=0
Jul 31 15:22:34.717678 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.717589    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl" event={"ID":"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22","Type":"ContainerDied","Data":"546ac6deaec8f69eff36d0f31ffa4a0674125a26b61a792709008f35e529b9b1"}
Jul 31 15:22:34.717678 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.717658    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl" event={"ID":"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22","Type":"ContainerDied","Data":"501d97e83f836d9580bbd02622edfd50232566740283edb24f16b7a784a7d331"}
Jul 31 15:22:34.717783 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.717683    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="501d97e83f836d9580bbd02622edfd50232566740283edb24f16b7a784a7d331"
Jul 31 15:22:34.728885 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.728862    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:22:34.834821 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.834726    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxv7f\" (UniqueName: \"kubernetes.io/projected/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-kube-api-access-qxv7f\") pod \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") "
Jul 31 15:22:34.834821 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.834777    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-dshm\") pod \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") "
Jul 31 15:22:34.834821 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.834809    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-tmp-dir\") pod \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") "
Jul 31 15:22:34.835114 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.834835    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-tls-certs\") pod \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") "
Jul 31 15:22:34.835114 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.834863    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-home\") pod \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") "
Jul 31 15:22:34.835114 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.834903    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-model-cache\") pod \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\" (UID: \"db1aeba4-9b76-4a2c-9d6e-3d21a7134e22\") "
Jul 31 15:22:34.835284 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.835114    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db1aeba4-9b76-4a2c-9d6e-3d21a7134e22" (UID: "db1aeba4-9b76-4a2c-9d6e-3d21a7134e22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:22:34.835284 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.835156    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-home" (OuterVolumeSpecName: "home") pod "db1aeba4-9b76-4a2c-9d6e-3d21a7134e22" (UID: "db1aeba4-9b76-4a2c-9d6e-3d21a7134e22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:22:34.835284 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.835251    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-model-cache" (OuterVolumeSpecName: "model-cache") pod "db1aeba4-9b76-4a2c-9d6e-3d21a7134e22" (UID: "db1aeba4-9b76-4a2c-9d6e-3d21a7134e22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:22:34.837169 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.837124    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db1aeba4-9b76-4a2c-9d6e-3d21a7134e22" (UID: "db1aeba4-9b76-4a2c-9d6e-3d21a7134e22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:22:34.837169 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.837122    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-kube-api-access-qxv7f" (OuterVolumeSpecName: "kube-api-access-qxv7f") pod "db1aeba4-9b76-4a2c-9d6e-3d21a7134e22" (UID: "db1aeba4-9b76-4a2c-9d6e-3d21a7134e22"). InnerVolumeSpecName "kube-api-access-qxv7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:22:34.837366 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.837189    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-dshm" (OuterVolumeSpecName: "dshm") pod "db1aeba4-9b76-4a2c-9d6e-3d21a7134e22" (UID: "db1aeba4-9b76-4a2c-9d6e-3d21a7134e22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:22:34.936097 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.936055    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:22:34.936097 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.936095    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxv7f\" (UniqueName: \"kubernetes.io/projected/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-kube-api-access-qxv7f\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:22:34.936305 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.936111    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:22:34.936305 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.936124    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:22:34.936305 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.936134    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:22:34.936305 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:34.936146    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:22:35.720974 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:35.720940    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"
Jul 31 15:22:35.746561 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:35.746495    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"]
Jul 31 15:22:35.748881 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:35.748857    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-nwldl"]
Jul 31 15:22:36.490637 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:22:36.490596    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db1aeba4-9b76-4a2c-9d6e-3d21a7134e22" path="/var/lib/kubelet/pods/db1aeba4-9b76-4a2c-9d6e-3d21a7134e22/volumes"
Jul 31 15:23:09.425512 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:09.425471    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"]
Jul 31 15:23:09.425981 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:09.425935    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db1aeba4-9b76-4a2c-9d6e-3d21a7134e22" containerName="main"
Jul 31 15:23:09.425981 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:09.425956    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="db1aeba4-9b76-4a2c-9d6e-3d21a7134e22" containerName="main"
Jul 31 15:23:09.426059 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:09.426043    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="db1aeba4-9b76-4a2c-9d6e-3d21a7134e22" containerName="main"
Jul 31 15:23:09.429572 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:09.429553    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"
Jul 31 15:23:09.433618 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:09.433593    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-sg6v2\""
Jul 31 15:23:09.433618 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:09.433612    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 31 15:23:09.441150 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:09.441125    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"]
Jul 31 15:23:09.539765 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:09.539730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q64j\" (UniqueName: \"kubernetes.io/projected/11e682c3-b899-474a-a957-6ce3fa0bf26f-kube-api-access-8q64j\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4\" (UID: \"11e682c3-b899-474a-a957-6ce3fa0bf26f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"
Jul 31 15:23:09.539948 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:09.539783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e682c3-b899-474a-a957-6ce3fa0bf26f-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4\" (UID: \"11e682c3-b899-474a-a957-6ce3fa0bf26f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"
Jul 31 15:23:09.640805 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:09.640769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q64j\" (UniqueName: \"kubernetes.io/projected/11e682c3-b899-474a-a957-6ce3fa0bf26f-kube-api-access-8q64j\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4\" (UID: \"11e682c3-b899-474a-a957-6ce3fa0bf26f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"
Jul 31 15:23:09.640998 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:09.640858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e682c3-b899-474a-a957-6ce3fa0bf26f-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4\" (UID: \"11e682c3-b899-474a-a957-6ce3fa0bf26f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"
Jul 31 15:23:09.643372 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:09.643333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e682c3-b899-474a-a957-6ce3fa0bf26f-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4\" (UID: \"11e682c3-b899-474a-a957-6ce3fa0bf26f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"
Jul 31 15:23:09.649251 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:09.649224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q64j\" (UniqueName: \"kubernetes.io/projected/11e682c3-b899-474a-a957-6ce3fa0bf26f-kube-api-access-8q64j\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4\" (UID: \"11e682c3-b899-474a-a957-6ce3fa0bf26f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"
Jul 31 15:23:09.742512 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:09.742479    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"
Jul 31 15:23:09.876388 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:09.876341    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"]
Jul 31 15:23:09.878496 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:23:09.878467    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e682c3_b899_474a_a957_6ce3fa0bf26f.slice/crio-edc561e20970b1fcaef4996fe597a9fcc92d8ef12b3ae4ae45516a8de6ac97df WatchSource:0}: Error finding container edc561e20970b1fcaef4996fe597a9fcc92d8ef12b3ae4ae45516a8de6ac97df: Status 404 returned error can't find the container with id edc561e20970b1fcaef4996fe597a9fcc92d8ef12b3ae4ae45516a8de6ac97df
Jul 31 15:23:10.855881 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:10.855844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4" event={"ID":"11e682c3-b899-474a-a957-6ce3fa0bf26f","Type":"ContainerStarted","Data":"461bde7ad2eac88a211c1abc2ff5a8964b1a33ed80118d1ee757e5b6aa60e9fb"}
Jul 31 15:23:10.856316 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:10.855900    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4" event={"ID":"11e682c3-b899-474a-a957-6ce3fa0bf26f","Type":"ContainerStarted","Data":"edc561e20970b1fcaef4996fe597a9fcc92d8ef12b3ae4ae45516a8de6ac97df"}
Jul 31 15:23:10.856316 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:10.855980    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"
Jul 31 15:23:10.874949 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:10.874886    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4" podStartSLOduration=1.8748528599999998 podStartE2EDuration="1.87485286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:23:10.87372565 +0000 UTC m=+2088.882937520" watchObservedRunningTime="2026-07-31 15:23:10.87485286 +0000 UTC m=+2088.884064730"
Jul 31 15:23:41.861616 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:41.861582    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"
Jul 31 15:23:59.818288 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:23:59.818254    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"]
Jul 31 15:24:01.861822 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:01.861781    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4" podUID="11e682c3-b899-474a-a957-6ce3fa0bf26f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 15:24:11.861377 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:11.861313    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4" podUID="11e682c3-b899-474a-a957-6ce3fa0bf26f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 15:24:14.837177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:14.837093    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4" podUID="11e682c3-b899-474a-a957-6ce3fa0bf26f" containerName="main" containerID="cri-o://461bde7ad2eac88a211c1abc2ff5a8964b1a33ed80118d1ee757e5b6aa60e9fb" gracePeriod=45
Jul 31 15:24:15.075950 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.075922    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"
Jul 31 15:24:15.085696 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.085677    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q64j\" (UniqueName: \"kubernetes.io/projected/11e682c3-b899-474a-a957-6ce3fa0bf26f-kube-api-access-8q64j\") pod \"11e682c3-b899-474a-a957-6ce3fa0bf26f\" (UID: \"11e682c3-b899-474a-a957-6ce3fa0bf26f\") "
Jul 31 15:24:15.085757 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.085745    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e682c3-b899-474a-a957-6ce3fa0bf26f-tls-certs\") pod \"11e682c3-b899-474a-a957-6ce3fa0bf26f\" (UID: \"11e682c3-b899-474a-a957-6ce3fa0bf26f\") "
Jul 31 15:24:15.087853 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.087801    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11e682c3-b899-474a-a957-6ce3fa0bf26f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "11e682c3-b899-474a-a957-6ce3fa0bf26f" (UID: "11e682c3-b899-474a-a957-6ce3fa0bf26f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:24:15.087923 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.087897    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e682c3-b899-474a-a957-6ce3fa0bf26f-kube-api-access-8q64j" (OuterVolumeSpecName: "kube-api-access-8q64j") pod "11e682c3-b899-474a-a957-6ce3fa0bf26f" (UID: "11e682c3-b899-474a-a957-6ce3fa0bf26f"). InnerVolumeSpecName "kube-api-access-8q64j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:24:15.100399 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.100342    2578 generic.go:356] "Generic (PLEG): container finished" podID="11e682c3-b899-474a-a957-6ce3fa0bf26f" containerID="461bde7ad2eac88a211c1abc2ff5a8964b1a33ed80118d1ee757e5b6aa60e9fb" exitCode=0
Jul 31 15:24:15.100511 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.100391    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4" event={"ID":"11e682c3-b899-474a-a957-6ce3fa0bf26f","Type":"ContainerDied","Data":"461bde7ad2eac88a211c1abc2ff5a8964b1a33ed80118d1ee757e5b6aa60e9fb"}
Jul 31 15:24:15.100511 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.100455    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4" event={"ID":"11e682c3-b899-474a-a957-6ce3fa0bf26f","Type":"ContainerDied","Data":"edc561e20970b1fcaef4996fe597a9fcc92d8ef12b3ae4ae45516a8de6ac97df"}
Jul 31 15:24:15.100511 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.100478    2578 scope.go:117] "RemoveContainer" containerID="461bde7ad2eac88a211c1abc2ff5a8964b1a33ed80118d1ee757e5b6aa60e9fb"
Jul 31 15:24:15.100633 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.100407    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"
Jul 31 15:24:15.113782 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.111196    2578 scope.go:117] "RemoveContainer" containerID="461bde7ad2eac88a211c1abc2ff5a8964b1a33ed80118d1ee757e5b6aa60e9fb"
Jul 31 15:24:15.113782 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:24:15.113021    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"461bde7ad2eac88a211c1abc2ff5a8964b1a33ed80118d1ee757e5b6aa60e9fb\": container with ID starting with 461bde7ad2eac88a211c1abc2ff5a8964b1a33ed80118d1ee757e5b6aa60e9fb not found: ID does not exist" containerID="461bde7ad2eac88a211c1abc2ff5a8964b1a33ed80118d1ee757e5b6aa60e9fb"
Jul 31 15:24:15.113782 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.113056    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461bde7ad2eac88a211c1abc2ff5a8964b1a33ed80118d1ee757e5b6aa60e9fb"} err="failed to get container status \"461bde7ad2eac88a211c1abc2ff5a8964b1a33ed80118d1ee757e5b6aa60e9fb\": rpc error: code = NotFound desc = could not find container \"461bde7ad2eac88a211c1abc2ff5a8964b1a33ed80118d1ee757e5b6aa60e9fb\": container with ID starting with 461bde7ad2eac88a211c1abc2ff5a8964b1a33ed80118d1ee757e5b6aa60e9fb not found: ID does not exist"
Jul 31 15:24:15.134893 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.134865    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"]
Jul 31 15:24:15.140097 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.140066    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemntw4"]
Jul 31 15:24:15.186416 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.186384    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e682c3-b899-474a-a957-6ce3fa0bf26f-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:24:15.186416 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.186417    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q64j\" (UniqueName: \"kubernetes.io/projected/11e682c3-b899-474a-a957-6ce3fa0bf26f-kube-api-access-8q64j\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:24:15.592238 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.592203    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"]
Jul 31 15:24:15.600464 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:24:15.600434    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:24:15.600464 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:24:15.600464 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:24:15.600464 ip-10-0-133-32 kubenswrapper[2578]:  > containerID="eb2de1b37d29b23a67fdafc847daaa9811dbe69daf41f26d176a2d5f250ea0af" cmd=["/bin/sleep","15"]
Jul 31 15:24:15.600758 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.600483    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9" message="" err=<
Jul 31 15:24:15.600758 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:24:15.600758 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:24:15.600758 ip-10-0-133-32 kubenswrapper[2578]:  >
Jul 31 15:24:15.600758 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:24:15.600495    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:24:15.600758 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:24:15.600758 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:24:15.600758 ip-10-0-133-32 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9" podUID="27d6e8b1-6c19-4882-95bc-701f98b87f59" containerName="main" containerID="cri-o://eb2de1b37d29b23a67fdafc847daaa9811dbe69daf41f26d176a2d5f250ea0af"
Jul 31 15:24:15.600758 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.600538    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9" podUID="27d6e8b1-6c19-4882-95bc-701f98b87f59" containerName="main" containerID="cri-o://eb2de1b37d29b23a67fdafc847daaa9811dbe69daf41f26d176a2d5f250ea0af" gracePeriod=60
Jul 31 15:24:15.649556 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.649510    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9" podUID="27d6e8b1-6c19-4882-95bc-701f98b87f59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 31 15:24:15.938570 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.938543    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:24:15.992749 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.992711    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-model-cache\") pod \"27d6e8b1-6c19-4882-95bc-701f98b87f59\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") "
Jul 31 15:24:15.992910 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.992758    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-tmp-dir\") pod \"27d6e8b1-6c19-4882-95bc-701f98b87f59\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") "
Jul 31 15:24:15.992910 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.992788    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-dshm\") pod \"27d6e8b1-6c19-4882-95bc-701f98b87f59\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") "
Jul 31 15:24:15.992910 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.992818    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8mxq\" (UniqueName: \"kubernetes.io/projected/27d6e8b1-6c19-4882-95bc-701f98b87f59-kube-api-access-q8mxq\") pod \"27d6e8b1-6c19-4882-95bc-701f98b87f59\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") "
Jul 31 15:24:15.992910 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.992860    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-home\") pod \"27d6e8b1-6c19-4882-95bc-701f98b87f59\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") "
Jul 31 15:24:15.992910 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.992887    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27d6e8b1-6c19-4882-95bc-701f98b87f59-tls-certs\") pod \"27d6e8b1-6c19-4882-95bc-701f98b87f59\" (UID: \"27d6e8b1-6c19-4882-95bc-701f98b87f59\") "
Jul 31 15:24:15.993155 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.992983    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27d6e8b1-6c19-4882-95bc-701f98b87f59" (UID: "27d6e8b1-6c19-4882-95bc-701f98b87f59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:15.993155 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.993021    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-model-cache" (OuterVolumeSpecName: "model-cache") pod "27d6e8b1-6c19-4882-95bc-701f98b87f59" (UID: "27d6e8b1-6c19-4882-95bc-701f98b87f59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:15.993155 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.993114    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-home" (OuterVolumeSpecName: "home") pod "27d6e8b1-6c19-4882-95bc-701f98b87f59" (UID: "27d6e8b1-6c19-4882-95bc-701f98b87f59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:15.993155 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.993139    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:24:15.993340 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.993180    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:24:15.994992 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.994961    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-dshm" (OuterVolumeSpecName: "dshm") pod "27d6e8b1-6c19-4882-95bc-701f98b87f59" (UID: "27d6e8b1-6c19-4882-95bc-701f98b87f59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:15.995101 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.994999    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d6e8b1-6c19-4882-95bc-701f98b87f59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27d6e8b1-6c19-4882-95bc-701f98b87f59" (UID: "27d6e8b1-6c19-4882-95bc-701f98b87f59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:24:15.995101 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:15.995048    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d6e8b1-6c19-4882-95bc-701f98b87f59-kube-api-access-q8mxq" (OuterVolumeSpecName: "kube-api-access-q8mxq") pod "27d6e8b1-6c19-4882-95bc-701f98b87f59" (UID: "27d6e8b1-6c19-4882-95bc-701f98b87f59"). InnerVolumeSpecName "kube-api-access-q8mxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:24:16.093867 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:16.093833    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:24:16.093867 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:16.093862    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8mxq\" (UniqueName: \"kubernetes.io/projected/27d6e8b1-6c19-4882-95bc-701f98b87f59-kube-api-access-q8mxq\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:24:16.093867 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:16.093873    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27d6e8b1-6c19-4882-95bc-701f98b87f59-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:24:16.094112 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:16.093881    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27d6e8b1-6c19-4882-95bc-701f98b87f59-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:24:16.106445 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:16.106339    2578 generic.go:356] "Generic (PLEG): container finished" podID="27d6e8b1-6c19-4882-95bc-701f98b87f59" containerID="eb2de1b37d29b23a67fdafc847daaa9811dbe69daf41f26d176a2d5f250ea0af" exitCode=0
Jul 31 15:24:16.106445 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:16.106393    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9" event={"ID":"27d6e8b1-6c19-4882-95bc-701f98b87f59","Type":"ContainerDied","Data":"eb2de1b37d29b23a67fdafc847daaa9811dbe69daf41f26d176a2d5f250ea0af"}
Jul 31 15:24:16.106609 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:16.106472    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9" event={"ID":"27d6e8b1-6c19-4882-95bc-701f98b87f59","Type":"ContainerDied","Data":"99b50e3a0e1d23bfbf1caa3694e06604b4d7dd14e31f50228f0e2d8ca95afc30"}
Jul 31 15:24:16.106609 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:16.106408    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"
Jul 31 15:24:16.106609 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:16.106507    2578 scope.go:117] "RemoveContainer" containerID="eb2de1b37d29b23a67fdafc847daaa9811dbe69daf41f26d176a2d5f250ea0af"
Jul 31 15:24:16.116690 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:16.116578    2578 scope.go:117] "RemoveContainer" containerID="eb2de1b37d29b23a67fdafc847daaa9811dbe69daf41f26d176a2d5f250ea0af"
Jul 31 15:24:16.117003 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:24:16.116973    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb2de1b37d29b23a67fdafc847daaa9811dbe69daf41f26d176a2d5f250ea0af\": container with ID starting with eb2de1b37d29b23a67fdafc847daaa9811dbe69daf41f26d176a2d5f250ea0af not found: ID does not exist" containerID="eb2de1b37d29b23a67fdafc847daaa9811dbe69daf41f26d176a2d5f250ea0af"
Jul 31 15:24:16.117117 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:16.117011    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb2de1b37d29b23a67fdafc847daaa9811dbe69daf41f26d176a2d5f250ea0af"} err="failed to get container status \"eb2de1b37d29b23a67fdafc847daaa9811dbe69daf41f26d176a2d5f250ea0af\": rpc error: code = NotFound desc = could not find container \"eb2de1b37d29b23a67fdafc847daaa9811dbe69daf41f26d176a2d5f250ea0af\": container with ID starting with eb2de1b37d29b23a67fdafc847daaa9811dbe69daf41f26d176a2d5f250ea0af not found: ID does not exist"
Jul 31 15:24:16.136844 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:16.136817    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"]
Jul 31 15:24:16.140692 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:16.140656    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hcfn9"]
Jul 31 15:24:16.488993 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:16.488960    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11e682c3-b899-474a-a957-6ce3fa0bf26f" path="/var/lib/kubelet/pods/11e682c3-b899-474a-a957-6ce3fa0bf26f/volumes"
Jul 31 15:24:16.489401 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:16.489383    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27d6e8b1-6c19-4882-95bc-701f98b87f59" path="/var/lib/kubelet/pods/27d6e8b1-6c19-4882-95bc-701f98b87f59/volumes"
Jul 31 15:24:39.405420 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.405384    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"]
Jul 31 15:24:39.407664 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.405642    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11e682c3-b899-474a-a957-6ce3fa0bf26f" containerName="main"
Jul 31 15:24:39.407664 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.405652    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="11e682c3-b899-474a-a957-6ce3fa0bf26f" containerName="main"
Jul 31 15:24:39.407664 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.405665    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d6e8b1-6c19-4882-95bc-701f98b87f59" containerName="main"
Jul 31 15:24:39.407664 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.405671    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d6e8b1-6c19-4882-95bc-701f98b87f59" containerName="main"
Jul 31 15:24:39.407664 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.405719    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="11e682c3-b899-474a-a957-6ce3fa0bf26f" containerName="main"
Jul 31 15:24:39.407664 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.405727    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="27d6e8b1-6c19-4882-95bc-701f98b87f59" containerName="main"
Jul 31 15:24:39.408544 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.408527    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.412455 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.412432    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Jul 31 15:24:39.412611 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.412465    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Jul 31 15:24:39.412611 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.412584    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-48cmm\""
Jul 31 15:24:39.412746 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.412683    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Jul 31 15:24:39.421339 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.421308    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"]
Jul 31 15:24:39.468012 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.467971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e09121e5-5db3-41c2-af5b-798130c062eb-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.468012 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.468023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.468247 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.468092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-home\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.468247 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.468121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.468247 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.468165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgwj9\" (UniqueName: \"kubernetes.io/projected/e09121e5-5db3-41c2-af5b-798130c062eb-kube-api-access-dgwj9\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.468247 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.468195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-dshm\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.569621 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.569584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgwj9\" (UniqueName: \"kubernetes.io/projected/e09121e5-5db3-41c2-af5b-798130c062eb-kube-api-access-dgwj9\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.569621 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.569626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-dshm\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.569895 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.569665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e09121e5-5db3-41c2-af5b-798130c062eb-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.569895 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.569691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.569895 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.569714    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-home\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.569895 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.569730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.570178 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.570152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.570254 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.570165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.570254 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.570225    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-home\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.571921 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.571899    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-dshm\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.572199 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.572179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e09121e5-5db3-41c2-af5b-798130c062eb-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.579010 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.578990    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgwj9\" (UniqueName: \"kubernetes.io/projected/e09121e5-5db3-41c2-af5b-798130c062eb-kube-api-access-dgwj9\") pod \"delw-v1-kserve-6cf8cbb847-d2dxm\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.720922 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.720878    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:39.850080 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:39.850053    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"]
Jul 31 15:24:39.851891 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:24:39.851865    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09121e5_5db3_41c2_af5b_798130c062eb.slice/crio-b3a2fd69f0d8c4056619af544637bbcd78577faf96d3063042c815e02414ea05 WatchSource:0}: Error finding container b3a2fd69f0d8c4056619af544637bbcd78577faf96d3063042c815e02414ea05: Status 404 returned error can't find the container with id b3a2fd69f0d8c4056619af544637bbcd78577faf96d3063042c815e02414ea05
Jul 31 15:24:40.198726 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:40.198628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm" event={"ID":"e09121e5-5db3-41c2-af5b-798130c062eb","Type":"ContainerStarted","Data":"b4c3ad6b245f1fa9adcee3c23811e40dae272d77e69f7576f7fc60c486cbae29"}
Jul 31 15:24:40.198726 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:40.198700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm" event={"ID":"e09121e5-5db3-41c2-af5b-798130c062eb","Type":"ContainerStarted","Data":"b3a2fd69f0d8c4056619af544637bbcd78577faf96d3063042c815e02414ea05"}
Jul 31 15:24:40.220477 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:40.220422    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm" podStartSLOduration=1.220406984 podStartE2EDuration="1.220406984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:24:40.218293022 +0000 UTC m=+2178.227504916" watchObservedRunningTime="2026-07-31 15:24:40.220406984 +0000 UTC m=+2178.229618853"
Jul 31 15:24:40.722001 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:40.721960    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:49.721874 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:49.721828    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:49.733949 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:49.733921    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:24:50.249613 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:24:50.249585    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:25:04.809459 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:04.809423    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"]
Jul 31 15:25:04.812838 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:04.812819    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:04.816065 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:04.816042    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 31 15:25:04.816232 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:04.816070    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-sdx99\""
Jul 31 15:25:04.816394 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:04.816373    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:25:04.816394 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:04.816377    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Jul 31 15:25:04.816583 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:04.816502    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 31 15:25:04.816583 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:04.816567    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 31 15:25:04.826151 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:04.826122    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"]
Jul 31 15:25:04.990073 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:04.990033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8e1066-713a-4265-a278-1dd59bc60142-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:04.990073 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:04.990076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:04.990306 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:04.990095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:04.990306 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:04.990174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:04.990306 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:04.990210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:04.990306 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:04.990253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba8e1066-713a-4265-a278-1dd59bc60142-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:04.990306 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:04.990293    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:04.990498 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:04.990317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8cx8\" (UniqueName: \"kubernetes.io/projected/ba8e1066-713a-4265-a278-1dd59bc60142-kube-api-access-c8cx8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.091449 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.091329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8e1066-713a-4265-a278-1dd59bc60142-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.091449 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.091392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.091449 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.091422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.091736 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.091460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.091736 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.091490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.091736 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.091533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba8e1066-713a-4265-a278-1dd59bc60142-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.091736 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.091580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.091736 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.091608    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8cx8\" (UniqueName: \"kubernetes.io/projected/ba8e1066-713a-4265-a278-1dd59bc60142-kube-api-access-c8cx8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.092000 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.091900    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.092000 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.091936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.092000 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.091966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.092173 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.092152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.092392 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.092366    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba8e1066-713a-4265-a278-1dd59bc60142-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.093803 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.093781    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.094038 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.094020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8e1066-713a-4265-a278-1dd59bc60142-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.100808 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.100775    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8cx8\" (UniqueName: \"kubernetes.io/projected/ba8e1066-713a-4265-a278-1dd59bc60142-kube-api-access-c8cx8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.123747 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.123705    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:05.260873 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.260846    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"]
Jul 31 15:25:05.263742 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:25:05.263717    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8e1066_713a_4265_a278_1dd59bc60142.slice/crio-ebbedcf5953026bf4085ed90eabd6307afde98d8bbab7996d354e7a1dabe5046 WatchSource:0}: Error finding container ebbedcf5953026bf4085ed90eabd6307afde98d8bbab7996d354e7a1dabe5046: Status 404 returned error can't find the container with id ebbedcf5953026bf4085ed90eabd6307afde98d8bbab7996d354e7a1dabe5046
Jul 31 15:25:05.294327 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:05.294297    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc" event={"ID":"ba8e1066-713a-4265-a278-1dd59bc60142","Type":"ContainerStarted","Data":"ebbedcf5953026bf4085ed90eabd6307afde98d8bbab7996d354e7a1dabe5046"}
Jul 31 15:25:06.300189 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:06.300151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc" event={"ID":"ba8e1066-713a-4265-a278-1dd59bc60142","Type":"ContainerStarted","Data":"2f8b316975efd1393fc8741fcb1de96a5db308b5cd21870f117cc94cc4812448"}
Jul 31 15:25:09.314031 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:09.313997    2578 generic.go:356] "Generic (PLEG): container finished" podID="ba8e1066-713a-4265-a278-1dd59bc60142" containerID="2f8b316975efd1393fc8741fcb1de96a5db308b5cd21870f117cc94cc4812448" exitCode=0
Jul 31 15:25:09.314031 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:09.314038    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc" event={"ID":"ba8e1066-713a-4265-a278-1dd59bc60142","Type":"ContainerDied","Data":"2f8b316975efd1393fc8741fcb1de96a5db308b5cd21870f117cc94cc4812448"}
Jul 31 15:25:10.319856 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:10.319820    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc" event={"ID":"ba8e1066-713a-4265-a278-1dd59bc60142","Type":"ContainerStarted","Data":"07ce1a3b89d9fd0744087bd8cedf0093af4d447c61711eaed2dcb9f0b9222a3b"}
Jul 31 15:25:10.345023 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:10.344973    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc" podStartSLOduration=3.249329157 podStartE2EDuration="6.344957547s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.09562839s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:25:10.34298532 +0000 UTC m=+2208.352197180" watchObservedRunningTime="2026-07-31 15:25:10.344957547 +0000 UTC m=+2208.354169415"
Jul 31 15:25:11.124765 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:11.124721    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:15.124689 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:15.124652    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:25:15.125986 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:15.125958    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc" podUID="ba8e1066-713a-4265-a278-1dd59bc60142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 31 15:25:25.124288 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:25.124240    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc" podUID="ba8e1066-713a-4265-a278-1dd59bc60142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 31 15:25:35.124790 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:35.124734    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc" podUID="ba8e1066-713a-4265-a278-1dd59bc60142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 31 15:25:45.124651 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:45.124546    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc" podUID="ba8e1066-713a-4265-a278-1dd59bc60142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 31 15:25:55.124779 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:25:55.124728    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc" podUID="ba8e1066-713a-4265-a278-1dd59bc60142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 31 15:26:05.125087 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:05.125040    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc" podUID="ba8e1066-713a-4265-a278-1dd59bc60142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 31 15:26:12.456126 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:12.456080    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"]
Jul 31 15:26:15.124560 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:15.124514    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc" podUID="ba8e1066-713a-4265-a278-1dd59bc60142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 31 15:26:25.135066 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:25.135024    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:26:25.147798 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:25.147763    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:26:27.474900 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.474861    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm" podUID="e09121e5-5db3-41c2-af5b-798130c062eb" containerName="main" containerID="cri-o://b4c3ad6b245f1fa9adcee3c23811e40dae272d77e69f7576f7fc60c486cbae29" gracePeriod=45
Jul 31 15:26:27.670758 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.670719    2578 generic.go:356] "Generic (PLEG): container finished" podID="e09121e5-5db3-41c2-af5b-798130c062eb" containerID="b4c3ad6b245f1fa9adcee3c23811e40dae272d77e69f7576f7fc60c486cbae29" exitCode=0
Jul 31 15:26:27.670955 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.670795    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm" event={"ID":"e09121e5-5db3-41c2-af5b-798130c062eb","Type":"ContainerDied","Data":"b4c3ad6b245f1fa9adcee3c23811e40dae272d77e69f7576f7fc60c486cbae29"}
Jul 31 15:26:27.725761 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.725733    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:26:27.830264 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.830235    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-dshm\") pod \"e09121e5-5db3-41c2-af5b-798130c062eb\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") "
Jul 31 15:26:27.830428 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.830309    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-tmp-dir\") pod \"e09121e5-5db3-41c2-af5b-798130c062eb\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") "
Jul 31 15:26:27.830428 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.830330    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e09121e5-5db3-41c2-af5b-798130c062eb-tls-certs\") pod \"e09121e5-5db3-41c2-af5b-798130c062eb\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") "
Jul 31 15:26:27.830428 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.830371    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-home\") pod \"e09121e5-5db3-41c2-af5b-798130c062eb\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") "
Jul 31 15:26:27.830428 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.830405    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgwj9\" (UniqueName: \"kubernetes.io/projected/e09121e5-5db3-41c2-af5b-798130c062eb-kube-api-access-dgwj9\") pod \"e09121e5-5db3-41c2-af5b-798130c062eb\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") "
Jul 31 15:26:27.830604 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.830440    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-model-cache\") pod \"e09121e5-5db3-41c2-af5b-798130c062eb\" (UID: \"e09121e5-5db3-41c2-af5b-798130c062eb\") "
Jul 31 15:26:27.830654 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.830612    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e09121e5-5db3-41c2-af5b-798130c062eb" (UID: "e09121e5-5db3-41c2-af5b-798130c062eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:26:27.830654 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.830626    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-home" (OuterVolumeSpecName: "home") pod "e09121e5-5db3-41c2-af5b-798130c062eb" (UID: "e09121e5-5db3-41c2-af5b-798130c062eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:26:27.830848 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.830820    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "e09121e5-5db3-41c2-af5b-798130c062eb" (UID: "e09121e5-5db3-41c2-af5b-798130c062eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:26:27.832583 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.832556    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e09121e5-5db3-41c2-af5b-798130c062eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e09121e5-5db3-41c2-af5b-798130c062eb" (UID: "e09121e5-5db3-41c2-af5b-798130c062eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:26:27.832676 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.832588    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-dshm" (OuterVolumeSpecName: "dshm") pod "e09121e5-5db3-41c2-af5b-798130c062eb" (UID: "e09121e5-5db3-41c2-af5b-798130c062eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:26:27.832676 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.832612    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09121e5-5db3-41c2-af5b-798130c062eb-kube-api-access-dgwj9" (OuterVolumeSpecName: "kube-api-access-dgwj9") pod "e09121e5-5db3-41c2-af5b-798130c062eb" (UID: "e09121e5-5db3-41c2-af5b-798130c062eb"). InnerVolumeSpecName "kube-api-access-dgwj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:26:27.931628 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.931575    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:26:27.931628 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.931620    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e09121e5-5db3-41c2-af5b-798130c062eb-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:26:27.931628 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.931631    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:26:27.931628 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.931639    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgwj9\" (UniqueName: \"kubernetes.io/projected/e09121e5-5db3-41c2-af5b-798130c062eb-kube-api-access-dgwj9\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:26:27.931628 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.931648    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:26:27.932051 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:27.931659    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e09121e5-5db3-41c2-af5b-798130c062eb-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:26:28.676814 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:28.676780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm" event={"ID":"e09121e5-5db3-41c2-af5b-798130c062eb","Type":"ContainerDied","Data":"b3a2fd69f0d8c4056619af544637bbcd78577faf96d3063042c815e02414ea05"}
Jul 31 15:26:28.677240 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:28.676825    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"
Jul 31 15:26:28.677240 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:28.676844    2578 scope.go:117] "RemoveContainer" containerID="b4c3ad6b245f1fa9adcee3c23811e40dae272d77e69f7576f7fc60c486cbae29"
Jul 31 15:26:28.694265 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:28.694238    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"]
Jul 31 15:26:28.698826 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:28.698795    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-d2dxm"]
Jul 31 15:26:30.489805 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:26:30.489766    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e09121e5-5db3-41c2-af5b-798130c062eb" path="/var/lib/kubelet/pods/e09121e5-5db3-41c2-af5b-798130c062eb/volumes"
Jul 31 15:27:22.697127 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:22.697052    2578 scope.go:117] "RemoveContainer" containerID="546ac6deaec8f69eff36d0f31ffa4a0674125a26b61a792709008f35e529b9b1"
Jul 31 15:27:23.015669 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:23.015627    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"]
Jul 31 15:27:38.037926 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.037882    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc" podUID="ba8e1066-713a-4265-a278-1dd59bc60142" containerName="main" containerID="cri-o://07ce1a3b89d9fd0744087bd8cedf0093af4d447c61711eaed2dcb9f0b9222a3b" gracePeriod=45
Jul 31 15:27:38.136038 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.135983    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc" podUID="ba8e1066-713a-4265-a278-1dd59bc60142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 31 15:27:38.688200 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.688174    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:27:38.830120 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.830083    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-kserve-provision-location\") pod \"ba8e1066-713a-4265-a278-1dd59bc60142\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") "
Jul 31 15:27:38.830120 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.830121    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-model-cache\") pod \"ba8e1066-713a-4265-a278-1dd59bc60142\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") "
Jul 31 15:27:38.830392 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.830164    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-dshm\") pod \"ba8e1066-713a-4265-a278-1dd59bc60142\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") "
Jul 31 15:27:38.830392 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.830187    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-home\") pod \"ba8e1066-713a-4265-a278-1dd59bc60142\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") "
Jul 31 15:27:38.830392 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.830223    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8e1066-713a-4265-a278-1dd59bc60142-tls-certs\") pod \"ba8e1066-713a-4265-a278-1dd59bc60142\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") "
Jul 31 15:27:38.830392 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.830238    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-tmp-dir\") pod \"ba8e1066-713a-4265-a278-1dd59bc60142\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") "
Jul 31 15:27:38.830392 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.830261    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8cx8\" (UniqueName: \"kubernetes.io/projected/ba8e1066-713a-4265-a278-1dd59bc60142-kube-api-access-c8cx8\") pod \"ba8e1066-713a-4265-a278-1dd59bc60142\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") "
Jul 31 15:27:38.830392 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.830288    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba8e1066-713a-4265-a278-1dd59bc60142-cabundle-cert\") pod \"ba8e1066-713a-4265-a278-1dd59bc60142\" (UID: \"ba8e1066-713a-4265-a278-1dd59bc60142\") "
Jul 31 15:27:38.830709 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.830471    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba8e1066-713a-4265-a278-1dd59bc60142" (UID: "ba8e1066-713a-4265-a278-1dd59bc60142"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:27:38.830709 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.830584    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:27:38.830709 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.830649    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-home" (OuterVolumeSpecName: "home") pod "ba8e1066-713a-4265-a278-1dd59bc60142" (UID: "ba8e1066-713a-4265-a278-1dd59bc60142"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:27:38.830866 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.830825    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba8e1066-713a-4265-a278-1dd59bc60142-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ba8e1066-713a-4265-a278-1dd59bc60142" (UID: "ba8e1066-713a-4265-a278-1dd59bc60142"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:27:38.832756 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.832726    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8e1066-713a-4265-a278-1dd59bc60142-kube-api-access-c8cx8" (OuterVolumeSpecName: "kube-api-access-c8cx8") pod "ba8e1066-713a-4265-a278-1dd59bc60142" (UID: "ba8e1066-713a-4265-a278-1dd59bc60142"). InnerVolumeSpecName "kube-api-access-c8cx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:27:38.832756 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.832744    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba8e1066-713a-4265-a278-1dd59bc60142-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba8e1066-713a-4265-a278-1dd59bc60142" (UID: "ba8e1066-713a-4265-a278-1dd59bc60142"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:27:38.832954 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.832787    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-dshm" (OuterVolumeSpecName: "dshm") pod "ba8e1066-713a-4265-a278-1dd59bc60142" (UID: "ba8e1066-713a-4265-a278-1dd59bc60142"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:27:38.842038 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.842011    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba8e1066-713a-4265-a278-1dd59bc60142" (UID: "ba8e1066-713a-4265-a278-1dd59bc60142"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:27:38.853511 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.853481    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ba8e1066-713a-4265-a278-1dd59bc60142" (UID: "ba8e1066-713a-4265-a278-1dd59bc60142"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:27:38.931719 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.931688    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8e1066-713a-4265-a278-1dd59bc60142-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:27:38.931719 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.931716    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:27:38.931719 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.931726    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8cx8\" (UniqueName: \"kubernetes.io/projected/ba8e1066-713a-4265-a278-1dd59bc60142-kube-api-access-c8cx8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:27:38.931932 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.931735    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba8e1066-713a-4265-a278-1dd59bc60142-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:27:38.931932 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.931749    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:27:38.931932 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.931760    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:27:38.931932 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.931768    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba8e1066-713a-4265-a278-1dd59bc60142-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:27:38.969302 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.969269    2578 generic.go:356] "Generic (PLEG): container finished" podID="ba8e1066-713a-4265-a278-1dd59bc60142" containerID="07ce1a3b89d9fd0744087bd8cedf0093af4d447c61711eaed2dcb9f0b9222a3b" exitCode=0
Jul 31 15:27:38.969463 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.969316    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"
Jul 31 15:27:38.969463 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.969379    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc" event={"ID":"ba8e1066-713a-4265-a278-1dd59bc60142","Type":"ContainerDied","Data":"07ce1a3b89d9fd0744087bd8cedf0093af4d447c61711eaed2dcb9f0b9222a3b"}
Jul 31 15:27:38.969463 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.969434    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc" event={"ID":"ba8e1066-713a-4265-a278-1dd59bc60142","Type":"ContainerDied","Data":"ebbedcf5953026bf4085ed90eabd6307afde98d8bbab7996d354e7a1dabe5046"}
Jul 31 15:27:38.969463 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.969458    2578 scope.go:117] "RemoveContainer" containerID="07ce1a3b89d9fd0744087bd8cedf0093af4d447c61711eaed2dcb9f0b9222a3b"
Jul 31 15:27:38.978113 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.978095    2578 scope.go:117] "RemoveContainer" containerID="2f8b316975efd1393fc8741fcb1de96a5db308b5cd21870f117cc94cc4812448"
Jul 31 15:27:38.985849 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.985830    2578 scope.go:117] "RemoveContainer" containerID="07ce1a3b89d9fd0744087bd8cedf0093af4d447c61711eaed2dcb9f0b9222a3b"
Jul 31 15:27:38.986092 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:27:38.986073    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ce1a3b89d9fd0744087bd8cedf0093af4d447c61711eaed2dcb9f0b9222a3b\": container with ID starting with 07ce1a3b89d9fd0744087bd8cedf0093af4d447c61711eaed2dcb9f0b9222a3b not found: ID does not exist" containerID="07ce1a3b89d9fd0744087bd8cedf0093af4d447c61711eaed2dcb9f0b9222a3b"
Jul 31 15:27:38.986177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.986104    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ce1a3b89d9fd0744087bd8cedf0093af4d447c61711eaed2dcb9f0b9222a3b"} err="failed to get container status \"07ce1a3b89d9fd0744087bd8cedf0093af4d447c61711eaed2dcb9f0b9222a3b\": rpc error: code = NotFound desc = could not find container \"07ce1a3b89d9fd0744087bd8cedf0093af4d447c61711eaed2dcb9f0b9222a3b\": container with ID starting with 07ce1a3b89d9fd0744087bd8cedf0093af4d447c61711eaed2dcb9f0b9222a3b not found: ID does not exist"
Jul 31 15:27:38.986177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.986127    2578 scope.go:117] "RemoveContainer" containerID="2f8b316975efd1393fc8741fcb1de96a5db308b5cd21870f117cc94cc4812448"
Jul 31 15:27:38.986385 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:27:38.986367    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f8b316975efd1393fc8741fcb1de96a5db308b5cd21870f117cc94cc4812448\": container with ID starting with 2f8b316975efd1393fc8741fcb1de96a5db308b5cd21870f117cc94cc4812448 not found: ID does not exist" containerID="2f8b316975efd1393fc8741fcb1de96a5db308b5cd21870f117cc94cc4812448"
Jul 31 15:27:38.986435 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.986391    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f8b316975efd1393fc8741fcb1de96a5db308b5cd21870f117cc94cc4812448"} err="failed to get container status \"2f8b316975efd1393fc8741fcb1de96a5db308b5cd21870f117cc94cc4812448\": rpc error: code = NotFound desc = could not find container \"2f8b316975efd1393fc8741fcb1de96a5db308b5cd21870f117cc94cc4812448\": container with ID starting with 2f8b316975efd1393fc8741fcb1de96a5db308b5cd21870f117cc94cc4812448 not found: ID does not exist"
Jul 31 15:27:38.993624 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.993600    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"]
Jul 31 15:27:38.998824 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:38.998805    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-796886f8f6-6vssc"]
Jul 31 15:27:40.489139 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:40.489098    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba8e1066-713a-4265-a278-1dd59bc60142" path="/var/lib/kubelet/pods/ba8e1066-713a-4265-a278-1dd59bc60142/volumes"
Jul 31 15:27:48.586957 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.586916    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"]
Jul 31 15:27:48.587473 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.587180    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba8e1066-713a-4265-a278-1dd59bc60142" containerName="storage-initializer"
Jul 31 15:27:48.587473 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.587190    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba8e1066-713a-4265-a278-1dd59bc60142" containerName="storage-initializer"
Jul 31 15:27:48.587473 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.587201    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e09121e5-5db3-41c2-af5b-798130c062eb" containerName="main"
Jul 31 15:27:48.587473 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.587207    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e09121e5-5db3-41c2-af5b-798130c062eb" containerName="main"
Jul 31 15:27:48.587473 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.587223    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba8e1066-713a-4265-a278-1dd59bc60142" containerName="main"
Jul 31 15:27:48.587473 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.587228    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba8e1066-713a-4265-a278-1dd59bc60142" containerName="main"
Jul 31 15:27:48.587473 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.587271    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e09121e5-5db3-41c2-af5b-798130c062eb" containerName="main"
Jul 31 15:27:48.587473 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.587279    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba8e1066-713a-4265-a278-1dd59bc60142" containerName="main"
Jul 31 15:27:48.592082 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.592060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.594919 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.594896    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Jul 31 15:27:48.595496 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.595470    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 31 15:27:48.595620 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.595523    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:27:48.595620 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.595546    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-wc4nt\""
Jul 31 15:27:48.595721 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.595702    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 31 15:27:48.596120 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.596103    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 31 15:27:48.601115 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.601092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6fcf0e2-cc77-4752-b0df-ce9f59033427-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.601206 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.601135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.601256 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.601217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.601313 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.601254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc4fp\" (UniqueName: \"kubernetes.io/projected/d6fcf0e2-cc77-4752-b0df-ce9f59033427-kube-api-access-dc4fp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.601313 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.601277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.601449 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.601337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.601449 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.601433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6fcf0e2-cc77-4752-b0df-ce9f59033427-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.601556 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.601493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.604960 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.604936    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"]
Jul 31 15:27:48.701834 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.701799    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6fcf0e2-cc77-4752-b0df-ce9f59033427-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.702003 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.701842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.702003 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.701960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6fcf0e2-cc77-4752-b0df-ce9f59033427-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.702113 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.701998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.702113 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.702057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.702113 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.702082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4fp\" (UniqueName: \"kubernetes.io/projected/d6fcf0e2-cc77-4752-b0df-ce9f59033427-kube-api-access-dc4fp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.702272 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.702108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.702272 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.702143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.702429 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.702408    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.702491 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.702446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.702560 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.702541    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.702632 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.702612    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.702876 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.702855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6fcf0e2-cc77-4752-b0df-ce9f59033427-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.704175 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.704157    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.704276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.704259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6fcf0e2-cc77-4752-b0df-ce9f59033427-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.712136 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.712113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4fp\" (UniqueName: \"kubernetes.io/projected/d6fcf0e2-cc77-4752-b0df-ce9f59033427-kube-api-access-dc4fp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:48.903130 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:48.903042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:49.045456 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:49.045417    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"]
Jul 31 15:27:49.048889 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:27:49.048857    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6fcf0e2_cc77_4752_b0df_ce9f59033427.slice/crio-3b47047566d267b6527ea063db021560095da71704cb7c23bca08fd9f9a7760a WatchSource:0}: Error finding container 3b47047566d267b6527ea063db021560095da71704cb7c23bca08fd9f9a7760a: Status 404 returned error can't find the container with id 3b47047566d267b6527ea063db021560095da71704cb7c23bca08fd9f9a7760a
Jul 31 15:27:50.013410 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:50.013366    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2" event={"ID":"d6fcf0e2-cc77-4752-b0df-ce9f59033427","Type":"ContainerStarted","Data":"c6b646d340cd091a7e1979876e5737a4b30bc52bb92e128fce528cde16f57706"}
Jul 31 15:27:50.013881 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:50.013430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2" event={"ID":"d6fcf0e2-cc77-4752-b0df-ce9f59033427","Type":"ContainerStarted","Data":"3b47047566d267b6527ea063db021560095da71704cb7c23bca08fd9f9a7760a"}
Jul 31 15:27:53.025935 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:53.025901    2578 generic.go:356] "Generic (PLEG): container finished" podID="d6fcf0e2-cc77-4752-b0df-ce9f59033427" containerID="c6b646d340cd091a7e1979876e5737a4b30bc52bb92e128fce528cde16f57706" exitCode=0
Jul 31 15:27:53.026342 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:53.025948    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2" event={"ID":"d6fcf0e2-cc77-4752-b0df-ce9f59033427","Type":"ContainerDied","Data":"c6b646d340cd091a7e1979876e5737a4b30bc52bb92e128fce528cde16f57706"}
Jul 31 15:27:54.032521 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:54.032479    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2" event={"ID":"d6fcf0e2-cc77-4752-b0df-ce9f59033427","Type":"ContainerStarted","Data":"d834662db747d2a155b4d5b42c4050ff210b1caecb7d12c6ab6e3b83b7b63105"}
Jul 31 15:27:54.058570 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:54.058511    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2" podStartSLOduration=2.909475339 podStartE2EDuration="6.058494494s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.149019155s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:27:54.056617628 +0000 UTC m=+2372.065829526" watchObservedRunningTime="2026-07-31 15:27:54.058494494 +0000 UTC m=+2372.067706363"
Jul 31 15:27:54.904193 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:54.904157    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:58.903552 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:58.903513    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:27:58.905158 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:27:58.905124    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2" podUID="d6fcf0e2-cc77-4752-b0df-ce9f59033427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 31 15:28:08.904522 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:08.904471    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2" podUID="d6fcf0e2-cc77-4752-b0df-ce9f59033427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 31 15:28:18.903802 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:18.903743    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2" podUID="d6fcf0e2-cc77-4752-b0df-ce9f59033427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 31 15:28:23.906941 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:23.906902    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"]
Jul 31 15:28:23.911833 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:23.911805    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:23.914811 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:23.914789    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Jul 31 15:28:23.916121 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:23.916089    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Jul 31 15:28:23.916249 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:23.916123    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-n7v8j\""
Jul 31 15:28:23.916249 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:23.916141    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Jul 31 15:28:23.922775 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:23.922749    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"]
Jul 31 15:28:23.996770 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:23.996737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-model-cache\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:23.996950 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:23.996786    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/329c1c70-1bd3-4dfe-9f18-be09217b51f2-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:23.996950 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:23.996864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-home\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:23.996950 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:23.996905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-dshm\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:23.996950 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:23.996938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:23.997096 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:23.996962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngp8v\" (UniqueName: \"kubernetes.io/projected/329c1c70-1bd3-4dfe-9f18-be09217b51f2-kube-api-access-ngp8v\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:24.098078 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:24.098034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-model-cache\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:24.098276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:24.098084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/329c1c70-1bd3-4dfe-9f18-be09217b51f2-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:24.098276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:24.098119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-home\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:24.098276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:24.098144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-dshm\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:24.098276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:24.098179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:24.098276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:24.098204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngp8v\" (UniqueName: \"kubernetes.io/projected/329c1c70-1bd3-4dfe-9f18-be09217b51f2-kube-api-access-ngp8v\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:24.098593 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:24.098491    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-model-cache\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:24.098593 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:24.098562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-home\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:24.098760 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:24.098739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:24.100664 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:24.100639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-dshm\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:24.100817 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:24.100799    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/329c1c70-1bd3-4dfe-9f18-be09217b51f2-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:24.107615 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:24.107587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngp8v\" (UniqueName: \"kubernetes.io/projected/329c1c70-1bd3-4dfe-9f18-be09217b51f2-kube-api-access-ngp8v\") pod \"owner-v1-kserve-75579ff5bc-ttchp\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:24.223039 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:24.223000    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:24.362626 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:24.362547    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"]
Jul 31 15:28:24.366048 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:28:24.366013    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329c1c70_1bd3_4dfe_9f18_be09217b51f2.slice/crio-439b829a37845d7607483527886307db4c542a426631611ebbf9e1e9ab7d98af WatchSource:0}: Error finding container 439b829a37845d7607483527886307db4c542a426631611ebbf9e1e9ab7d98af: Status 404 returned error can't find the container with id 439b829a37845d7607483527886307db4c542a426631611ebbf9e1e9ab7d98af
Jul 31 15:28:25.164666 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:25.164628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp" event={"ID":"329c1c70-1bd3-4dfe-9f18-be09217b51f2","Type":"ContainerStarted","Data":"05b12ec603fce654c20f6957d31f2d123f8f59e3c4d6d706612ab2d3de3dc472"}
Jul 31 15:28:25.165061 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:25.164709    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp" event={"ID":"329c1c70-1bd3-4dfe-9f18-be09217b51f2","Type":"ContainerStarted","Data":"439b829a37845d7607483527886307db4c542a426631611ebbf9e1e9ab7d98af"}
Jul 31 15:28:25.188087 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:25.188038    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp" podStartSLOduration=2.18802448 podStartE2EDuration="2.18802448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:28:25.18670998 +0000 UTC m=+2403.195921861" watchObservedRunningTime="2026-07-31 15:28:25.18802448 +0000 UTC m=+2403.197236349"
Jul 31 15:28:25.223963 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:25.223929    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:28.903983 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:28.903935    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2" podUID="d6fcf0e2-cc77-4752-b0df-ce9f59033427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 31 15:28:34.223430 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:34.223392    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:34.236000 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:34.235971    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:35.223548 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:35.223518    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:28:38.903874 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:38.903816    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2" podUID="d6fcf0e2-cc77-4752-b0df-ce9f59033427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 31 15:28:48.904377 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:48.904272    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2" podUID="d6fcf0e2-cc77-4752-b0df-ce9f59033427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 31 15:28:58.904367 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:28:58.904298    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2" podUID="d6fcf0e2-cc77-4752-b0df-ce9f59033427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 31 15:29:08.915006 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:08.914965    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:29:08.923014 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:08.922991    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:29:11.126891 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:11.126848    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"]
Jul 31 15:29:22.086438 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:22.086401    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"]
Jul 31 15:29:26.148514 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.148463    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2" podUID="d6fcf0e2-cc77-4752-b0df-ce9f59033427" containerName="main" containerID="cri-o://d834662db747d2a155b4d5b42c4050ff210b1caecb7d12c6ab6e3b83b7b63105" gracePeriod=45
Jul 31 15:29:26.797889 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.797860    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:29:26.846441 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.846412    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-kserve-provision-location\") pod \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") "
Jul 31 15:29:26.846441 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.846444    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-home\") pod \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") "
Jul 31 15:29:26.846675 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.846461    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dc4fp\" (UniqueName: \"kubernetes.io/projected/d6fcf0e2-cc77-4752-b0df-ce9f59033427-kube-api-access-dc4fp\") pod \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") "
Jul 31 15:29:26.846675 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.846524    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-tmp-dir\") pod \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") "
Jul 31 15:29:26.846675 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.846543    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6fcf0e2-cc77-4752-b0df-ce9f59033427-cabundle-cert\") pod \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") "
Jul 31 15:29:26.846675 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.846562    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6fcf0e2-cc77-4752-b0df-ce9f59033427-tls-certs\") pod \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") "
Jul 31 15:29:26.846675 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.846596    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-dshm\") pod \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") "
Jul 31 15:29:26.847040 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.846926    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-home" (OuterVolumeSpecName: "home") pod "d6fcf0e2-cc77-4752-b0df-ce9f59033427" (UID: "d6fcf0e2-cc77-4752-b0df-ce9f59033427"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:29:26.847040 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.846929    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6fcf0e2-cc77-4752-b0df-ce9f59033427-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d6fcf0e2-cc77-4752-b0df-ce9f59033427" (UID: "d6fcf0e2-cc77-4752-b0df-ce9f59033427"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:29:26.847040 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.846969    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-model-cache\") pod \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\" (UID: \"d6fcf0e2-cc77-4752-b0df-ce9f59033427\") "
Jul 31 15:29:26.847194 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.847054    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6fcf0e2-cc77-4752-b0df-ce9f59033427" (UID: "d6fcf0e2-cc77-4752-b0df-ce9f59033427"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:29:26.847255 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.847216    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:29:26.847255 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.847246    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6fcf0e2-cc77-4752-b0df-ce9f59033427-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:29:26.847375 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.847260    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:29:26.847375 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.847300    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6fcf0e2-cc77-4752-b0df-ce9f59033427" (UID: "d6fcf0e2-cc77-4752-b0df-ce9f59033427"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:29:26.848961 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.848929    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-dshm" (OuterVolumeSpecName: "dshm") pod "d6fcf0e2-cc77-4752-b0df-ce9f59033427" (UID: "d6fcf0e2-cc77-4752-b0df-ce9f59033427"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:29:26.849196 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.849163    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6fcf0e2-cc77-4752-b0df-ce9f59033427-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6fcf0e2-cc77-4752-b0df-ce9f59033427" (UID: "d6fcf0e2-cc77-4752-b0df-ce9f59033427"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:29:26.849196 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.849176    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6fcf0e2-cc77-4752-b0df-ce9f59033427-kube-api-access-dc4fp" (OuterVolumeSpecName: "kube-api-access-dc4fp") pod "d6fcf0e2-cc77-4752-b0df-ce9f59033427" (UID: "d6fcf0e2-cc77-4752-b0df-ce9f59033427"). InnerVolumeSpecName "kube-api-access-dc4fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:29:26.868848 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.868820    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6fcf0e2-cc77-4752-b0df-ce9f59033427" (UID: "d6fcf0e2-cc77-4752-b0df-ce9f59033427"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:29:26.948275 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.948210    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:29:26.948275 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.948237    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc4fp\" (UniqueName: \"kubernetes.io/projected/d6fcf0e2-cc77-4752-b0df-ce9f59033427-kube-api-access-dc4fp\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:29:26.948275 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.948247    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6fcf0e2-cc77-4752-b0df-ce9f59033427-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:29:26.948275 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.948258    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:29:26.948275 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:26.948267    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6fcf0e2-cc77-4752-b0df-ce9f59033427-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:29:27.433776 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:27.433730    2578 generic.go:356] "Generic (PLEG): container finished" podID="d6fcf0e2-cc77-4752-b0df-ce9f59033427" containerID="d834662db747d2a155b4d5b42c4050ff210b1caecb7d12c6ab6e3b83b7b63105" exitCode=0
Jul 31 15:29:27.434172 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:27.433789    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"
Jul 31 15:29:27.434172 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:27.433813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2" event={"ID":"d6fcf0e2-cc77-4752-b0df-ce9f59033427","Type":"ContainerDied","Data":"d834662db747d2a155b4d5b42c4050ff210b1caecb7d12c6ab6e3b83b7b63105"}
Jul 31 15:29:27.434172 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:27.433883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2" event={"ID":"d6fcf0e2-cc77-4752-b0df-ce9f59033427","Type":"ContainerDied","Data":"3b47047566d267b6527ea063db021560095da71704cb7c23bca08fd9f9a7760a"}
Jul 31 15:29:27.434172 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:27.433909    2578 scope.go:117] "RemoveContainer" containerID="d834662db747d2a155b4d5b42c4050ff210b1caecb7d12c6ab6e3b83b7b63105"
Jul 31 15:29:27.442856 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:27.442836    2578 scope.go:117] "RemoveContainer" containerID="c6b646d340cd091a7e1979876e5737a4b30bc52bb92e128fce528cde16f57706"
Jul 31 15:29:27.450866 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:27.450847    2578 scope.go:117] "RemoveContainer" containerID="d834662db747d2a155b4d5b42c4050ff210b1caecb7d12c6ab6e3b83b7b63105"
Jul 31 15:29:27.451093 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:29:27.451074    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d834662db747d2a155b4d5b42c4050ff210b1caecb7d12c6ab6e3b83b7b63105\": container with ID starting with d834662db747d2a155b4d5b42c4050ff210b1caecb7d12c6ab6e3b83b7b63105 not found: ID does not exist" containerID="d834662db747d2a155b4d5b42c4050ff210b1caecb7d12c6ab6e3b83b7b63105"
Jul 31 15:29:27.451161 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:27.451101    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d834662db747d2a155b4d5b42c4050ff210b1caecb7d12c6ab6e3b83b7b63105"} err="failed to get container status \"d834662db747d2a155b4d5b42c4050ff210b1caecb7d12c6ab6e3b83b7b63105\": rpc error: code = NotFound desc = could not find container \"d834662db747d2a155b4d5b42c4050ff210b1caecb7d12c6ab6e3b83b7b63105\": container with ID starting with d834662db747d2a155b4d5b42c4050ff210b1caecb7d12c6ab6e3b83b7b63105 not found: ID does not exist"
Jul 31 15:29:27.451161 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:27.451117    2578 scope.go:117] "RemoveContainer" containerID="c6b646d340cd091a7e1979876e5737a4b30bc52bb92e128fce528cde16f57706"
Jul 31 15:29:27.451364 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:29:27.451325    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b646d340cd091a7e1979876e5737a4b30bc52bb92e128fce528cde16f57706\": container with ID starting with c6b646d340cd091a7e1979876e5737a4b30bc52bb92e128fce528cde16f57706 not found: ID does not exist" containerID="c6b646d340cd091a7e1979876e5737a4b30bc52bb92e128fce528cde16f57706"
Jul 31 15:29:27.451412 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:27.451374    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b646d340cd091a7e1979876e5737a4b30bc52bb92e128fce528cde16f57706"} err="failed to get container status \"c6b646d340cd091a7e1979876e5737a4b30bc52bb92e128fce528cde16f57706\": rpc error: code = NotFound desc = could not find container \"c6b646d340cd091a7e1979876e5737a4b30bc52bb92e128fce528cde16f57706\": container with ID starting with c6b646d340cd091a7e1979876e5737a4b30bc52bb92e128fce528cde16f57706 not found: ID does not exist"
Jul 31 15:29:27.458545 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:27.458520    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"]
Jul 31 15:29:27.462599 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:27.462577    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6f7887cfff-2jjn2"]
Jul 31 15:29:28.489642 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:28.489612    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6fcf0e2-cc77-4752-b0df-ce9f59033427" path="/var/lib/kubelet/pods/d6fcf0e2-cc77-4752-b0df-ce9f59033427/volumes"
Jul 31 15:29:37.105098 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.105052    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp" podUID="329c1c70-1bd3-4dfe-9f18-be09217b51f2" containerName="main" containerID="cri-o://05b12ec603fce654c20f6957d31f2d123f8f59e3c4d6d706612ab2d3de3dc472" gracePeriod=45
Jul 31 15:29:37.212920 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.212879    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp" podUID="329c1c70-1bd3-4dfe-9f18-be09217b51f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Jul 31 15:29:37.346236 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.346211    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:29:37.427905 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.427806    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngp8v\" (UniqueName: \"kubernetes.io/projected/329c1c70-1bd3-4dfe-9f18-be09217b51f2-kube-api-access-ngp8v\") pod \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") "
Jul 31 15:29:37.427905 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.427890    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-dshm\") pod \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") "
Jul 31 15:29:37.428147 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.427939    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-model-cache\") pod \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") "
Jul 31 15:29:37.428147 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.427979    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-tmp-dir\") pod \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") "
Jul 31 15:29:37.428147 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.428017    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/329c1c70-1bd3-4dfe-9f18-be09217b51f2-tls-certs\") pod \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") "
Jul 31 15:29:37.428147 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.428037    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-home\") pod \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\" (UID: \"329c1c70-1bd3-4dfe-9f18-be09217b51f2\") "
Jul 31 15:29:37.428380 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.428288    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "329c1c70-1bd3-4dfe-9f18-be09217b51f2" (UID: "329c1c70-1bd3-4dfe-9f18-be09217b51f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:29:37.428380 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.428290    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "329c1c70-1bd3-4dfe-9f18-be09217b51f2" (UID: "329c1c70-1bd3-4dfe-9f18-be09217b51f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:29:37.428494 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.428441    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-home" (OuterVolumeSpecName: "home") pod "329c1c70-1bd3-4dfe-9f18-be09217b51f2" (UID: "329c1c70-1bd3-4dfe-9f18-be09217b51f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:29:37.430094 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.430063    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329c1c70-1bd3-4dfe-9f18-be09217b51f2-kube-api-access-ngp8v" (OuterVolumeSpecName: "kube-api-access-ngp8v") pod "329c1c70-1bd3-4dfe-9f18-be09217b51f2" (UID: "329c1c70-1bd3-4dfe-9f18-be09217b51f2"). InnerVolumeSpecName "kube-api-access-ngp8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:29:37.430207 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.430094    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-dshm" (OuterVolumeSpecName: "dshm") pod "329c1c70-1bd3-4dfe-9f18-be09217b51f2" (UID: "329c1c70-1bd3-4dfe-9f18-be09217b51f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:29:37.430207 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.430097    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329c1c70-1bd3-4dfe-9f18-be09217b51f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "329c1c70-1bd3-4dfe-9f18-be09217b51f2" (UID: "329c1c70-1bd3-4dfe-9f18-be09217b51f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:29:37.472834 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.472799    2578 generic.go:356] "Generic (PLEG): container finished" podID="329c1c70-1bd3-4dfe-9f18-be09217b51f2" containerID="05b12ec603fce654c20f6957d31f2d123f8f59e3c4d6d706612ab2d3de3dc472" exitCode=0
Jul 31 15:29:37.473006 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.472850    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"
Jul 31 15:29:37.473006 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.472882    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp" event={"ID":"329c1c70-1bd3-4dfe-9f18-be09217b51f2","Type":"ContainerDied","Data":"05b12ec603fce654c20f6957d31f2d123f8f59e3c4d6d706612ab2d3de3dc472"}
Jul 31 15:29:37.473006 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.472937    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp" event={"ID":"329c1c70-1bd3-4dfe-9f18-be09217b51f2","Type":"ContainerDied","Data":"439b829a37845d7607483527886307db4c542a426631611ebbf9e1e9ab7d98af"}
Jul 31 15:29:37.473006 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.472962    2578 scope.go:117] "RemoveContainer" containerID="05b12ec603fce654c20f6957d31f2d123f8f59e3c4d6d706612ab2d3de3dc472"
Jul 31 15:29:37.480781 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.480762    2578 scope.go:117] "RemoveContainer" containerID="05b12ec603fce654c20f6957d31f2d123f8f59e3c4d6d706612ab2d3de3dc472"
Jul 31 15:29:37.481100 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:29:37.481072    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b12ec603fce654c20f6957d31f2d123f8f59e3c4d6d706612ab2d3de3dc472\": container with ID starting with 05b12ec603fce654c20f6957d31f2d123f8f59e3c4d6d706612ab2d3de3dc472 not found: ID does not exist" containerID="05b12ec603fce654c20f6957d31f2d123f8f59e3c4d6d706612ab2d3de3dc472"
Jul 31 15:29:37.481187 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.481110    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b12ec603fce654c20f6957d31f2d123f8f59e3c4d6d706612ab2d3de3dc472"} err="failed to get container status \"05b12ec603fce654c20f6957d31f2d123f8f59e3c4d6d706612ab2d3de3dc472\": rpc error: code = NotFound desc = could not find container \"05b12ec603fce654c20f6957d31f2d123f8f59e3c4d6d706612ab2d3de3dc472\": container with ID starting with 05b12ec603fce654c20f6957d31f2d123f8f59e3c4d6d706612ab2d3de3dc472 not found: ID does not exist"
Jul 31 15:29:37.495565 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.495538    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"]
Jul 31 15:29:37.499332 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.499310    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-ttchp"]
Jul 31 15:29:37.528691 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.528650    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:29:37.528691 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.528693    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/329c1c70-1bd3-4dfe-9f18-be09217b51f2-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:29:37.528848 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.528706    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:29:37.528848 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.528719    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngp8v\" (UniqueName: \"kubernetes.io/projected/329c1c70-1bd3-4dfe-9f18-be09217b51f2-kube-api-access-ngp8v\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:29:37.528848 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.528732    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:29:37.528848 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:37.528746    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/329c1c70-1bd3-4dfe-9f18-be09217b51f2-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:29:38.488691 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:38.488655    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="329c1c70-1bd3-4dfe-9f18-be09217b51f2" path="/var/lib/kubelet/pods/329c1c70-1bd3-4dfe-9f18-be09217b51f2/volumes"
Jul 31 15:29:44.111477 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.110794    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"]
Jul 31 15:29:44.111477 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.111204    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6fcf0e2-cc77-4752-b0df-ce9f59033427" containerName="main"
Jul 31 15:29:44.111477 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.111225    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6fcf0e2-cc77-4752-b0df-ce9f59033427" containerName="main"
Jul 31 15:29:44.111477 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.111245    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6fcf0e2-cc77-4752-b0df-ce9f59033427" containerName="storage-initializer"
Jul 31 15:29:44.111477 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.111267    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6fcf0e2-cc77-4752-b0df-ce9f59033427" containerName="storage-initializer"
Jul 31 15:29:44.111477 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.111299    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="329c1c70-1bd3-4dfe-9f18-be09217b51f2" containerName="main"
Jul 31 15:29:44.111477 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.111309    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="329c1c70-1bd3-4dfe-9f18-be09217b51f2" containerName="main"
Jul 31 15:29:44.111477 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.111404    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="329c1c70-1bd3-4dfe-9f18-be09217b51f2" containerName="main"
Jul 31 15:29:44.111477 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.111417    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6fcf0e2-cc77-4752-b0df-ce9f59033427" containerName="main"
Jul 31 15:29:44.114455 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.114435    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.117476 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.117454    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Jul 31 15:29:44.117623 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.117594    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-jwldv\""
Jul 31 15:29:44.117705 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.117468    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Jul 31 15:29:44.117779 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.117479    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Jul 31 15:29:44.131620 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.131595    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"]
Jul 31 15:29:44.176463 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.176409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-istio-envoy\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.176655 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.176493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-credential-socket\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.176655 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.176524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-istio-data\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.176655 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.176540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-workload-socket\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.176655 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.176590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-workload-certs\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.176655 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.176624    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886c4367-2a51-4431-a847-4cafe8458840-istio-podinfo\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.176655 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.176643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886c4367-2a51-4431-a847-4cafe8458840-istio-token\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.176887 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.176672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj2cj\" (UniqueName: \"kubernetes.io/projected/886c4367-2a51-4431-a847-4cafe8458840-kube-api-access-mj2cj\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.176887 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.176702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886c4367-2a51-4431-a847-4cafe8458840-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.277266 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.277220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-istio-envoy\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.277266 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.277268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-credential-socket\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.277527 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.277394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-istio-data\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.277527 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.277426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-workload-socket\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.277527 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.277462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-workload-certs\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.277676 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.277550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886c4367-2a51-4431-a847-4cafe8458840-istio-podinfo\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.277676 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.277589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886c4367-2a51-4431-a847-4cafe8458840-istio-token\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.277676 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.277625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj2cj\" (UniqueName: \"kubernetes.io/projected/886c4367-2a51-4431-a847-4cafe8458840-kube-api-access-mj2cj\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.277676 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.277657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886c4367-2a51-4431-a847-4cafe8458840-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.277880 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.277736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-credential-socket\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.277880 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.277741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-workload-certs\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.277880 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.277783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-workload-socket\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.278062 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.278035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-istio-data\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.278424 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.278407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886c4367-2a51-4431-a847-4cafe8458840-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.279623 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.279602    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-istio-envoy\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.280137 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.280117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886c4367-2a51-4431-a847-4cafe8458840-istio-podinfo\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.288148 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.288121    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886c4367-2a51-4431-a847-4cafe8458840-istio-token\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.288234 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.288183    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj2cj\" (UniqueName: \"kubernetes.io/projected/886c4367-2a51-4431-a847-4cafe8458840-kube-api-access-mj2cj\") pod \"router-gateway-1-openshift-default-699c764795-sw7zr\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.424826 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.424734    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:44.567135 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.566980    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"]
Jul 31 15:29:44.569931 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:29:44.569900    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886c4367_2a51_4431_a847_4cafe8458840.slice/crio-b5f474de142fbf41f79496944ac58b4b54e0018b738916c349f32f2ac588ff45 WatchSource:0}: Error finding container b5f474de142fbf41f79496944ac58b4b54e0018b738916c349f32f2ac588ff45: Status 404 returned error can't find the container with id b5f474de142fbf41f79496944ac58b4b54e0018b738916c349f32f2ac588ff45
Jul 31 15:29:44.571953 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.571915    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:29:44.572114 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.571987    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:29:44.572114 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:44.572032    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:29:45.506961 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:45.506925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr" event={"ID":"886c4367-2a51-4431-a847-4cafe8458840","Type":"ContainerStarted","Data":"615923ffa0289675783fce6c4b7c4e6418b8ba54c009a54952123b86ced990e6"}
Jul 31 15:29:45.507340 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:45.507006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr" event={"ID":"886c4367-2a51-4431-a847-4cafe8458840","Type":"ContainerStarted","Data":"b5f474de142fbf41f79496944ac58b4b54e0018b738916c349f32f2ac588ff45"}
Jul 31 15:29:45.533472 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:45.533427    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr" podStartSLOduration=1.533412508 podStartE2EDuration="1.533412508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:29:45.531710029 +0000 UTC m=+2483.540921899" watchObservedRunningTime="2026-07-31 15:29:45.533412508 +0000 UTC m=+2483.542624413"
Jul 31 15:29:46.425491 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:46.425456    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:46.430234 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:46.430205    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:46.510997 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:46.510959    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:46.511856 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:46.511834    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:29:55.516187 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.516152    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"]
Jul 31 15:29:55.519446 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.519427    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.522187 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.522163    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Jul 31 15:29:55.522314 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.522188    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-hzhdm\""
Jul 31 15:29:55.522399 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.522311    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Jul 31 15:29:55.522399 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.522339    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Jul 31 15:29:55.522495 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.522407    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Jul 31 15:29:55.529684 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.529661    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"]
Jul 31 15:29:55.569023 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.568994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwrzd\" (UniqueName: \"kubernetes.io/projected/68e81c97-735e-42e2-baf1-7bec612b7e43-kube-api-access-zwrzd\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.569164 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.569038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-dshm\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.569164 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.569061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.569164 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.569143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-home\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.569333 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.569169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.569333 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.569220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e81c97-735e-42e2-baf1-7bec612b7e43-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.569333 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.569287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-model-cache\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.670298 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.670266    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e81c97-735e-42e2-baf1-7bec612b7e43-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.670487 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.670323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-model-cache\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.670487 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.670391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwrzd\" (UniqueName: \"kubernetes.io/projected/68e81c97-735e-42e2-baf1-7bec612b7e43-kube-api-access-zwrzd\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.670487 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.670422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-dshm\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.670665 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.670561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.670665 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.670634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-home\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.670776 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.670668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.670831 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.670803    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-model-cache\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.671052 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.671010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.671052 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.671034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.671466 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.671437    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-home\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.673096 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.673079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-dshm\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.673385 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.673363    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e81c97-735e-42e2-baf1-7bec612b7e43-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.679696 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.679667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwrzd\" (UniqueName: \"kubernetes.io/projected/68e81c97-735e-42e2-baf1-7bec612b7e43-kube-api-access-zwrzd\") pod \"e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.830953 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.830862    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:29:55.974392 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:55.974364    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"]
Jul 31 15:29:55.977149 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:29:55.977112    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e81c97_735e_42e2_baf1_7bec612b7e43.slice/crio-a188b7a40c131f42c35a7bb05155ab44b739d8c396a1576b64db220de44d29de WatchSource:0}: Error finding container a188b7a40c131f42c35a7bb05155ab44b739d8c396a1576b64db220de44d29de: Status 404 returned error can't find the container with id a188b7a40c131f42c35a7bb05155ab44b739d8c396a1576b64db220de44d29de
Jul 31 15:29:56.557564 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:56.557518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn" event={"ID":"68e81c97-735e-42e2-baf1-7bec612b7e43","Type":"ContainerStarted","Data":"3b030a00a6ed512ebc323c6513c1863452f4f8076e954c19ddc4ad25aac73e1b"}
Jul 31 15:29:56.558177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:29:56.557603    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn" event={"ID":"68e81c97-735e-42e2-baf1-7bec612b7e43","Type":"ContainerStarted","Data":"a188b7a40c131f42c35a7bb05155ab44b739d8c396a1576b64db220de44d29de"}
Jul 31 15:30:09.610701 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:09.610666    2578 generic.go:356] "Generic (PLEG): container finished" podID="68e81c97-735e-42e2-baf1-7bec612b7e43" containerID="3b030a00a6ed512ebc323c6513c1863452f4f8076e954c19ddc4ad25aac73e1b" exitCode=0
Jul 31 15:30:09.611111 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:09.610738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn" event={"ID":"68e81c97-735e-42e2-baf1-7bec612b7e43","Type":"ContainerDied","Data":"3b030a00a6ed512ebc323c6513c1863452f4f8076e954c19ddc4ad25aac73e1b"}
Jul 31 15:30:10.616639 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:10.616596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn" event={"ID":"68e81c97-735e-42e2-baf1-7bec612b7e43","Type":"ContainerStarted","Data":"380e765b187a671e520f78c9fe47bcee717c71df0d80cf707b6fb1f3280eee14"}
Jul 31 15:30:10.641086 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:10.641033    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn" podStartSLOduration=2.90772174 podStartE2EDuration="15.641017789s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.733296049s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:30:10.639542593 +0000 UTC m=+2508.648754463" watchObservedRunningTime="2026-07-31 15:30:10.641017789 +0000 UTC m=+2508.650229660"
Jul 31 15:30:10.831034 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:10.830989    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:30:15.831785 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:15.831691    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:30:15.833379 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:15.833337    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn" podUID="68e81c97-735e-42e2-baf1-7bec612b7e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 31 15:30:25.831723 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:25.831661    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn" podUID="68e81c97-735e-42e2-baf1-7bec612b7e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 31 15:30:35.832029 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:35.831983    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn" podUID="68e81c97-735e-42e2-baf1-7bec612b7e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 31 15:30:41.104694 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:41.104645    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"]
Jul 31 15:30:47.555852 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:47.555511    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr" podUID="886c4367-2a51-4431-a847-4cafe8458840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 31 15:30:49.780774 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:49.780742    2578 generic.go:356] "Generic (PLEG): container finished" podID="886c4367-2a51-4431-a847-4cafe8458840" containerID="615923ffa0289675783fce6c4b7c4e6418b8ba54c009a54952123b86ced990e6" exitCode=0
Jul 31 15:30:49.781215 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:49.780798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr" event={"ID":"886c4367-2a51-4431-a847-4cafe8458840","Type":"ContainerDied","Data":"615923ffa0289675783fce6c4b7c4e6418b8ba54c009a54952123b86ced990e6"}
Jul 31 15:30:49.781275 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:49.781221    2578 scope.go:117] "RemoveContainer" containerID="615923ffa0289675783fce6c4b7c4e6418b8ba54c009a54952123b86ced990e6"
Jul 31 15:30:49.782120 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:49.782086    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:30:49.782224 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:49.782158    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:30:49.782224 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:49.782188    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:30:50.787418 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:50.787384    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr" event={"ID":"886c4367-2a51-4431-a847-4cafe8458840","Type":"ContainerStarted","Data":"b84800e37dcd22409991a55be129021202e7765f4449907d4d2c2e5cbce44c4e"}
Jul 31 15:30:51.425296 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:51.425243    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:30:51.425546 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:51.425506    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr" podUID="886c4367-2a51-4431-a847-4cafe8458840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 31 15:30:52.425643 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:52.425601    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr" podUID="886c4367-2a51-4431-a847-4cafe8458840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 31 15:30:53.425511 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:53.425467    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr" podUID="886c4367-2a51-4431-a847-4cafe8458840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 31 15:30:54.425101 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:54.425057    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr" podUID="886c4367-2a51-4431-a847-4cafe8458840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 31 15:30:55.425911 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:55.425860    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr" podUID="886c4367-2a51-4431-a847-4cafe8458840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 31 15:30:56.127167 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:56.127128    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn" podUID="68e81c97-735e-42e2-baf1-7bec612b7e43" containerName="main" containerID="cri-o://380e765b187a671e520f78c9fe47bcee717c71df0d80cf707b6fb1f3280eee14" gracePeriod=45
Jul 31 15:30:56.429064 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:56.428979    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:30:56.429482 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:56.429456    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:30:56.430157 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:56.430132    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:30:57.961240 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:57.961214    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-config-deletion-blocked-when-referenced-0f029b85_e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn_68e81c97-735e-42e2-baf1-7bec612b7e43/main/0.log"
Jul 31 15:30:57.962848 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:57.962830    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:30:58.112764 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.112670    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-kserve-provision-location\") pod \"68e81c97-735e-42e2-baf1-7bec612b7e43\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") "
Jul 31 15:30:58.112764 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.112713    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-tmp-dir\") pod \"68e81c97-735e-42e2-baf1-7bec612b7e43\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") "
Jul 31 15:30:58.112764 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.112730    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-dshm\") pod \"68e81c97-735e-42e2-baf1-7bec612b7e43\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") "
Jul 31 15:30:58.112764 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.112766    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e81c97-735e-42e2-baf1-7bec612b7e43-tls-certs\") pod \"68e81c97-735e-42e2-baf1-7bec612b7e43\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") "
Jul 31 15:30:58.113105 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.112811    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-model-cache\") pod \"68e81c97-735e-42e2-baf1-7bec612b7e43\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") "
Jul 31 15:30:58.113105 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.112854    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwrzd\" (UniqueName: \"kubernetes.io/projected/68e81c97-735e-42e2-baf1-7bec612b7e43-kube-api-access-zwrzd\") pod \"68e81c97-735e-42e2-baf1-7bec612b7e43\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") "
Jul 31 15:30:58.113105 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.112898    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-home\") pod \"68e81c97-735e-42e2-baf1-7bec612b7e43\" (UID: \"68e81c97-735e-42e2-baf1-7bec612b7e43\") "
Jul 31 15:30:58.113105 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.113085    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68e81c97-735e-42e2-baf1-7bec612b7e43" (UID: "68e81c97-735e-42e2-baf1-7bec612b7e43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:30:58.113324 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.113104    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-model-cache" (OuterVolumeSpecName: "model-cache") pod "68e81c97-735e-42e2-baf1-7bec612b7e43" (UID: "68e81c97-735e-42e2-baf1-7bec612b7e43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:30:58.113324 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.113312    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-home" (OuterVolumeSpecName: "home") pod "68e81c97-735e-42e2-baf1-7bec612b7e43" (UID: "68e81c97-735e-42e2-baf1-7bec612b7e43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:30:58.115177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.115150    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-dshm" (OuterVolumeSpecName: "dshm") pod "68e81c97-735e-42e2-baf1-7bec612b7e43" (UID: "68e81c97-735e-42e2-baf1-7bec612b7e43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:30:58.115423 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.115394    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e81c97-735e-42e2-baf1-7bec612b7e43-kube-api-access-zwrzd" (OuterVolumeSpecName: "kube-api-access-zwrzd") pod "68e81c97-735e-42e2-baf1-7bec612b7e43" (UID: "68e81c97-735e-42e2-baf1-7bec612b7e43"). InnerVolumeSpecName "kube-api-access-zwrzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:30:58.115505 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.115459    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68e81c97-735e-42e2-baf1-7bec612b7e43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68e81c97-735e-42e2-baf1-7bec612b7e43" (UID: "68e81c97-735e-42e2-baf1-7bec612b7e43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:30:58.174499 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.174436    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "68e81c97-735e-42e2-baf1-7bec612b7e43" (UID: "68e81c97-735e-42e2-baf1-7bec612b7e43"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:30:58.213709 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.213670    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:30:58.213709 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.213705    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:30:58.213709 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.213717    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:30:58.213931 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.213726    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68e81c97-735e-42e2-baf1-7bec612b7e43-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:30:58.213931 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.213735    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:30:58.213931 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.213743    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwrzd\" (UniqueName: \"kubernetes.io/projected/68e81c97-735e-42e2-baf1-7bec612b7e43-kube-api-access-zwrzd\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:30:58.213931 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.213751    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68e81c97-735e-42e2-baf1-7bec612b7e43-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:30:58.821022 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.820944    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-config-deletion-blocked-when-referenced-0f029b85_e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn_68e81c97-735e-42e2-baf1-7bec612b7e43/main/0.log"
Jul 31 15:30:58.821330 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.821304    2578 generic.go:356] "Generic (PLEG): container finished" podID="68e81c97-735e-42e2-baf1-7bec612b7e43" containerID="380e765b187a671e520f78c9fe47bcee717c71df0d80cf707b6fb1f3280eee14" exitCode=130
Jul 31 15:30:58.821439 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.821376    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"
Jul 31 15:30:58.821439 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.821392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn" event={"ID":"68e81c97-735e-42e2-baf1-7bec612b7e43","Type":"ContainerDied","Data":"380e765b187a671e520f78c9fe47bcee717c71df0d80cf707b6fb1f3280eee14"}
Jul 31 15:30:58.821554 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.821464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn" event={"ID":"68e81c97-735e-42e2-baf1-7bec612b7e43","Type":"ContainerDied","Data":"a188b7a40c131f42c35a7bb05155ab44b739d8c396a1576b64db220de44d29de"}
Jul 31 15:30:58.821554 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.821501    2578 scope.go:117] "RemoveContainer" containerID="380e765b187a671e520f78c9fe47bcee717c71df0d80cf707b6fb1f3280eee14"
Jul 31 15:30:58.829485 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.829465    2578 scope.go:117] "RemoveContainer" containerID="3b030a00a6ed512ebc323c6513c1863452f4f8076e954c19ddc4ad25aac73e1b"
Jul 31 15:30:58.836982 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.836966    2578 scope.go:117] "RemoveContainer" containerID="380e765b187a671e520f78c9fe47bcee717c71df0d80cf707b6fb1f3280eee14"
Jul 31 15:30:58.837217 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:30:58.837196    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380e765b187a671e520f78c9fe47bcee717c71df0d80cf707b6fb1f3280eee14\": container with ID starting with 380e765b187a671e520f78c9fe47bcee717c71df0d80cf707b6fb1f3280eee14 not found: ID does not exist" containerID="380e765b187a671e520f78c9fe47bcee717c71df0d80cf707b6fb1f3280eee14"
Jul 31 15:30:58.837303 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.837223    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380e765b187a671e520f78c9fe47bcee717c71df0d80cf707b6fb1f3280eee14"} err="failed to get container status \"380e765b187a671e520f78c9fe47bcee717c71df0d80cf707b6fb1f3280eee14\": rpc error: code = NotFound desc = could not find container \"380e765b187a671e520f78c9fe47bcee717c71df0d80cf707b6fb1f3280eee14\": container with ID starting with 380e765b187a671e520f78c9fe47bcee717c71df0d80cf707b6fb1f3280eee14 not found: ID does not exist"
Jul 31 15:30:58.837303 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.837241    2578 scope.go:117] "RemoveContainer" containerID="3b030a00a6ed512ebc323c6513c1863452f4f8076e954c19ddc4ad25aac73e1b"
Jul 31 15:30:58.837513 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:30:58.837497    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b030a00a6ed512ebc323c6513c1863452f4f8076e954c19ddc4ad25aac73e1b\": container with ID starting with 3b030a00a6ed512ebc323c6513c1863452f4f8076e954c19ddc4ad25aac73e1b not found: ID does not exist" containerID="3b030a00a6ed512ebc323c6513c1863452f4f8076e954c19ddc4ad25aac73e1b"
Jul 31 15:30:58.837562 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.837517    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b030a00a6ed512ebc323c6513c1863452f4f8076e954c19ddc4ad25aac73e1b"} err="failed to get container status \"3b030a00a6ed512ebc323c6513c1863452f4f8076e954c19ddc4ad25aac73e1b\": rpc error: code = NotFound desc = could not find container \"3b030a00a6ed512ebc323c6513c1863452f4f8076e954c19ddc4ad25aac73e1b\": container with ID starting with 3b030a00a6ed512ebc323c6513c1863452f4f8076e954c19ddc4ad25aac73e1b not found: ID does not exist"
Jul 31 15:30:58.847585 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.847560    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"]
Jul 31 15:30:58.851693 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:30:58.851658    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6f9ff7fbdd-m64nn"]
Jul 31 15:31:00.489608 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:00.489573    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68e81c97-735e-42e2-baf1-7bec612b7e43" path="/var/lib/kubelet/pods/68e81c97-735e-42e2-baf1-7bec612b7e43/volumes"
Jul 31 15:31:06.023197 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.023164    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"]
Jul 31 15:31:06.023583 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.023474    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68e81c97-735e-42e2-baf1-7bec612b7e43" containerName="storage-initializer"
Jul 31 15:31:06.023583 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.023487    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="68e81c97-735e-42e2-baf1-7bec612b7e43" containerName="storage-initializer"
Jul 31 15:31:06.023583 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.023495    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68e81c97-735e-42e2-baf1-7bec612b7e43" containerName="main"
Jul 31 15:31:06.023583 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.023501    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="68e81c97-735e-42e2-baf1-7bec612b7e43" containerName="main"
Jul 31 15:31:06.023583 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.023542    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="68e81c97-735e-42e2-baf1-7bec612b7e43" containerName="main"
Jul 31 15:31:06.025471 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.025447    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.028268 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.028228    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 31 15:31:06.028405 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.028241    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 31 15:31:06.028405 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.028289    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 31 15:31:06.028405 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.028239    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 31 15:31:06.028405 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.028319    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:31:06.028639 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.028448    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-xpppn\""
Jul 31 15:31:06.038793 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.038771    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"]
Jul 31 15:31:06.178064 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.178028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-dshm\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.178238 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.178080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-home\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.178238 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.178152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-model-cache\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.178238 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.178182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.178238 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.178223    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73b55462-0f40-4b95-afa2-f1e71373cc38-cabundle-cert\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.178408 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.178240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-kserve-provision-location\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.178408 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.178261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmgxk\" (UniqueName: \"kubernetes.io/projected/73b55462-0f40-4b95-afa2-f1e71373cc38-kube-api-access-lmgxk\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.178408 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.178286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-tmp-dir\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.279019 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.278922    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73b55462-0f40-4b95-afa2-f1e71373cc38-cabundle-cert\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.279019 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.278966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-kserve-provision-location\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.279019 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.278987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmgxk\" (UniqueName: \"kubernetes.io/projected/73b55462-0f40-4b95-afa2-f1e71373cc38-kube-api-access-lmgxk\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.279019 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.279010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-tmp-dir\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.279370 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.279039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-dshm\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.279370 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.279070    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-home\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.279370 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.279100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-model-cache\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.279370 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.279118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.279603 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.279485    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-kserve-provision-location\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.279603 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.279537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-home\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.279603 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.279571    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-tmp-dir\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.279754 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.279735    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73b55462-0f40-4b95-afa2-f1e71373cc38-cabundle-cert\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.279810 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.279765    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-model-cache\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.281494 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.281470    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-dshm\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.281611 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.281594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.288152 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.288131    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmgxk\" (UniqueName: \"kubernetes.io/projected/73b55462-0f40-4b95-afa2-f1e71373cc38-kube-api-access-lmgxk\") pod \"stop-feature-test-kserve-7648cf7688-jrnfv\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.336258 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.336220    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:06.468716 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.468683    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"]
Jul 31 15:31:06.471879 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:31:06.471847    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b55462_0f40_4b95_afa2_f1e71373cc38.slice/crio-59e11d5000d8483e19ae3fdbd7edc8decef8bc55cca073120e01c1591ebec8e0 WatchSource:0}: Error finding container 59e11d5000d8483e19ae3fdbd7edc8decef8bc55cca073120e01c1591ebec8e0: Status 404 returned error can't find the container with id 59e11d5000d8483e19ae3fdbd7edc8decef8bc55cca073120e01c1591ebec8e0
Jul 31 15:31:06.853592 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.853546    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" event={"ID":"73b55462-0f40-4b95-afa2-f1e71373cc38","Type":"ContainerStarted","Data":"8556cc11d078c615aca078d097a38e780ff78c0968df6b746322bd118f6e6da2"}
Jul 31 15:31:06.853756 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:06.853617    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" event={"ID":"73b55462-0f40-4b95-afa2-f1e71373cc38","Type":"ContainerStarted","Data":"59e11d5000d8483e19ae3fdbd7edc8decef8bc55cca073120e01c1591ebec8e0"}
Jul 31 15:31:08.863300 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:08.863266    2578 generic.go:356] "Generic (PLEG): container finished" podID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerID="8556cc11d078c615aca078d097a38e780ff78c0968df6b746322bd118f6e6da2" exitCode=0
Jul 31 15:31:08.863707 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:08.863336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" event={"ID":"73b55462-0f40-4b95-afa2-f1e71373cc38","Type":"ContainerDied","Data":"8556cc11d078c615aca078d097a38e780ff78c0968df6b746322bd118f6e6da2"}
Jul 31 15:31:09.868657 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:09.868615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" event={"ID":"73b55462-0f40-4b95-afa2-f1e71373cc38","Type":"ContainerStarted","Data":"b85d2a62f4369071f9bb190fe5ddec803d56a549489445d428062903cbd783e8"}
Jul 31 15:31:09.894779 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:09.894714    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" podStartSLOduration=1.729308931 podStartE2EDuration="3.894693938s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.165385007s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:31:09.892130967 +0000 UTC m=+2567.901342818" watchObservedRunningTime="2026-07-31 15:31:09.894693938 +0000 UTC m=+2567.903905807"
Jul 31 15:31:10.337285 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:10.337243    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:16.336444 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:16.336404    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:31:16.337937 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:16.337902    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 31 15:31:26.336858 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:26.336808    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 31 15:31:36.336796 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:36.336749    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 31 15:31:46.337030 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:46.336928    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 31 15:31:56.337228 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:31:56.337174    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 31 15:32:06.337149 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:06.337096    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 31 15:32:09.973871 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:09.973833    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"]
Jul 31 15:32:09.974381 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:09.974099    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr" podUID="886c4367-2a51-4431-a847-4cafe8458840" containerName="istio-proxy" containerID="cri-o://b84800e37dcd22409991a55be129021202e7765f4449907d4d2c2e5cbce44c4e" gracePeriod=30
Jul 31 15:32:11.432116 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:11.432050    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr" podUID="886c4367-2a51-4431-a847-4cafe8458840" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:32:15.137292 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.137252    2578 generic.go:356] "Generic (PLEG): container finished" podID="886c4367-2a51-4431-a847-4cafe8458840" containerID="b84800e37dcd22409991a55be129021202e7765f4449907d4d2c2e5cbce44c4e" exitCode=0
Jul 31 15:32:15.137700 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.137315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr" event={"ID":"886c4367-2a51-4431-a847-4cafe8458840","Type":"ContainerDied","Data":"b84800e37dcd22409991a55be129021202e7765f4449907d4d2c2e5cbce44c4e"}
Jul 31 15:32:15.137700 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.137401    2578 scope.go:117] "RemoveContainer" containerID="615923ffa0289675783fce6c4b7c4e6418b8ba54c009a54952123b86ced990e6"
Jul 31 15:32:15.223234 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.223206    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:32:15.255679 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.255579    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-istio-data\") pod \"886c4367-2a51-4431-a847-4cafe8458840\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") "
Jul 31 15:32:15.255679 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.255628    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-istio-envoy\") pod \"886c4367-2a51-4431-a847-4cafe8458840\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") "
Jul 31 15:32:15.255911 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.255681    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886c4367-2a51-4431-a847-4cafe8458840-istio-podinfo\") pod \"886c4367-2a51-4431-a847-4cafe8458840\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") "
Jul 31 15:32:15.255911 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.255723    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-workload-socket\") pod \"886c4367-2a51-4431-a847-4cafe8458840\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") "
Jul 31 15:32:15.255911 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.255758    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886c4367-2a51-4431-a847-4cafe8458840-istio-token\") pod \"886c4367-2a51-4431-a847-4cafe8458840\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") "
Jul 31 15:32:15.255911 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.255783    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj2cj\" (UniqueName: \"kubernetes.io/projected/886c4367-2a51-4431-a847-4cafe8458840-kube-api-access-mj2cj\") pod \"886c4367-2a51-4431-a847-4cafe8458840\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") "
Jul 31 15:32:15.255911 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.255812    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-workload-certs\") pod \"886c4367-2a51-4431-a847-4cafe8458840\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") "
Jul 31 15:32:15.255911 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.255853    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886c4367-2a51-4431-a847-4cafe8458840-istiod-ca-cert\") pod \"886c4367-2a51-4431-a847-4cafe8458840\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") "
Jul 31 15:32:15.256241 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.256169    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-credential-socket\") pod \"886c4367-2a51-4431-a847-4cafe8458840\" (UID: \"886c4367-2a51-4431-a847-4cafe8458840\") "
Jul 31 15:32:15.257309 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.256170    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-istio-data" (OuterVolumeSpecName: "istio-data") pod "886c4367-2a51-4431-a847-4cafe8458840" (UID: "886c4367-2a51-4431-a847-4cafe8458840"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:32:15.257309 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.256292    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "886c4367-2a51-4431-a847-4cafe8458840" (UID: "886c4367-2a51-4431-a847-4cafe8458840"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:32:15.257309 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.256819    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/886c4367-2a51-4431-a847-4cafe8458840-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "886c4367-2a51-4431-a847-4cafe8458840" (UID: "886c4367-2a51-4431-a847-4cafe8458840"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:32:15.257309 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.257192    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "886c4367-2a51-4431-a847-4cafe8458840" (UID: "886c4367-2a51-4431-a847-4cafe8458840"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:32:15.257309 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.257278    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "886c4367-2a51-4431-a847-4cafe8458840" (UID: "886c4367-2a51-4431-a847-4cafe8458840"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:32:15.258963 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.258937    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "886c4367-2a51-4431-a847-4cafe8458840" (UID: "886c4367-2a51-4431-a847-4cafe8458840"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:32:15.258963 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.258950    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886c4367-2a51-4431-a847-4cafe8458840-istio-token" (OuterVolumeSpecName: "istio-token") pod "886c4367-2a51-4431-a847-4cafe8458840" (UID: "886c4367-2a51-4431-a847-4cafe8458840"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:32:15.259584 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.259548    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886c4367-2a51-4431-a847-4cafe8458840-kube-api-access-mj2cj" (OuterVolumeSpecName: "kube-api-access-mj2cj") pod "886c4367-2a51-4431-a847-4cafe8458840" (UID: "886c4367-2a51-4431-a847-4cafe8458840"). InnerVolumeSpecName "kube-api-access-mj2cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:32:15.260178 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.260144    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/886c4367-2a51-4431-a847-4cafe8458840-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "886c4367-2a51-4431-a847-4cafe8458840" (UID: "886c4367-2a51-4431-a847-4cafe8458840"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 15:32:15.356804 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.356766    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:32:15.356804 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.356795    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:32:15.356804 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.356805    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886c4367-2a51-4431-a847-4cafe8458840-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:32:15.356804 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.356814    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:32:15.357183 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.356822    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886c4367-2a51-4431-a847-4cafe8458840-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:32:15.357183 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.356836    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj2cj\" (UniqueName: \"kubernetes.io/projected/886c4367-2a51-4431-a847-4cafe8458840-kube-api-access-mj2cj\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:32:15.357183 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.356844    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:32:15.357183 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.356854    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886c4367-2a51-4431-a847-4cafe8458840-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:32:15.357183 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:15.356861    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886c4367-2a51-4431-a847-4cafe8458840-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:32:16.142607 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:16.142574    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"
Jul 31 15:32:16.143166 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:16.142572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr" event={"ID":"886c4367-2a51-4431-a847-4cafe8458840","Type":"ContainerDied","Data":"b5f474de142fbf41f79496944ac58b4b54e0018b738916c349f32f2ac588ff45"}
Jul 31 15:32:16.143166 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:16.142709    2578 scope.go:117] "RemoveContainer" containerID="b84800e37dcd22409991a55be129021202e7765f4449907d4d2c2e5cbce44c4e"
Jul 31 15:32:16.170062 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:16.170025    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"]
Jul 31 15:32:16.172378 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:16.172338    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sw7zr"]
Jul 31 15:32:16.337683 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:16.337637    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 31 15:32:16.489717 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:16.489682    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="886c4367-2a51-4431-a847-4cafe8458840" path="/var/lib/kubelet/pods/886c4367-2a51-4431-a847-4cafe8458840/volumes"
Jul 31 15:32:26.337822 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:26.337769    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 31 15:32:36.336784 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:36.336735    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 31 15:32:46.337150 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:46.337097    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 31 15:32:56.336784 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:32:56.336736    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 31 15:33:06.346925 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:06.346889    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:33:06.354801 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:06.354769    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:33:07.482815 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:07.482777    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"]
Jul 31 15:33:07.491072 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:33:07.491042    2578 secret.go:189] Couldn't get secret e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-self-signed-certs: secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 31 15:33:07.491224 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:33:07.491125    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs podName:73b55462-0f40-4b95-afa2-f1e71373cc38 nodeName:}" failed. No retries permitted until 2026-07-31 15:33:07.991105857 +0000 UTC m=+2686.000317708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs") pod "stop-feature-test-kserve-7648cf7688-jrnfv" (UID: "73b55462-0f40-4b95-afa2-f1e71373cc38") : secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 31 15:33:07.995831 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:33:07.995790    2578 secret.go:189] Couldn't get secret e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-self-signed-certs: secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 31 15:33:07.995998 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:33:07.995897    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs podName:73b55462-0f40-4b95-afa2-f1e71373cc38 nodeName:}" failed. No retries permitted until 2026-07-31 15:33:08.995874095 +0000 UTC m=+2687.005085958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs") pod "stop-feature-test-kserve-7648cf7688-jrnfv" (UID: "73b55462-0f40-4b95-afa2-f1e71373cc38") : secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 31 15:33:09.002034 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:33:09.001993    2578 secret.go:189] Couldn't get secret e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-self-signed-certs: secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 31 15:33:09.002559 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:33:09.002083    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs podName:73b55462-0f40-4b95-afa2-f1e71373cc38 nodeName:}" failed. No retries permitted until 2026-07-31 15:33:11.002063225 +0000 UTC m=+2689.011275072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs") pod "stop-feature-test-kserve-7648cf7688-jrnfv" (UID: "73b55462-0f40-4b95-afa2-f1e71373cc38") : secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 31 15:33:11.014416 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:33:11.014382    2578 secret.go:189] Couldn't get secret e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-self-signed-certs: secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 31 15:33:11.014875 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:33:11.014459    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs podName:73b55462-0f40-4b95-afa2-f1e71373cc38 nodeName:}" failed. No retries permitted until 2026-07-31 15:33:15.014441151 +0000 UTC m=+2693.023653012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs") pod "stop-feature-test-kserve-7648cf7688-jrnfv" (UID: "73b55462-0f40-4b95-afa2-f1e71373cc38") : secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 31 15:33:15.039832 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:33:15.039750    2578 secret.go:189] Couldn't get secret e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-self-signed-certs: secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 31 15:33:15.040208 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:33:15.039847    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs podName:73b55462-0f40-4b95-afa2-f1e71373cc38 nodeName:}" failed. No retries permitted until 2026-07-31 15:33:23.039829479 +0000 UTC m=+2701.049041330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs") pod "stop-feature-test-kserve-7648cf7688-jrnfv" (UID: "73b55462-0f40-4b95-afa2-f1e71373cc38") : secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 31 15:33:23.106976 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:33:23.106940    2578 secret.go:189] Couldn't get secret e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-self-signed-certs: secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 31 15:33:23.107408 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:33:23.107017    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs podName:73b55462-0f40-4b95-afa2-f1e71373cc38 nodeName:}" failed. No retries permitted until 2026-07-31 15:33:39.107001546 +0000 UTC m=+2717.116213398 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs") pod "stop-feature-test-kserve-7648cf7688-jrnfv" (UID: "73b55462-0f40-4b95-afa2-f1e71373cc38") : secret "stop-feature-test-kserve-self-signed-certs" not found
Jul 31 15:33:23.366597 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:23.366480    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerName="main" containerID="cri-o://b85d2a62f4369071f9bb190fe5ddec803d56a549489445d428062903cbd783e8" gracePeriod=45
Jul 31 15:33:24.109511 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.109485    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:33:24.216492 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.216399    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-dshm\") pod \"73b55462-0f40-4b95-afa2-f1e71373cc38\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") "
Jul 31 15:33:24.216492 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.216437    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmgxk\" (UniqueName: \"kubernetes.io/projected/73b55462-0f40-4b95-afa2-f1e71373cc38-kube-api-access-lmgxk\") pod \"73b55462-0f40-4b95-afa2-f1e71373cc38\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") "
Jul 31 15:33:24.216492 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.216472    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-model-cache\") pod \"73b55462-0f40-4b95-afa2-f1e71373cc38\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") "
Jul 31 15:33:24.216749 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.216528    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-kserve-provision-location\") pod \"73b55462-0f40-4b95-afa2-f1e71373cc38\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") "
Jul 31 15:33:24.216749 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.216552    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-tmp-dir\") pod \"73b55462-0f40-4b95-afa2-f1e71373cc38\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") "
Jul 31 15:33:24.216749 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.216594    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs\") pod \"73b55462-0f40-4b95-afa2-f1e71373cc38\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") "
Jul 31 15:33:24.216749 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.216627    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-home\") pod \"73b55462-0f40-4b95-afa2-f1e71373cc38\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") "
Jul 31 15:33:24.216749 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.216651    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73b55462-0f40-4b95-afa2-f1e71373cc38-cabundle-cert\") pod \"73b55462-0f40-4b95-afa2-f1e71373cc38\" (UID: \"73b55462-0f40-4b95-afa2-f1e71373cc38\") "
Jul 31 15:33:24.217014 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.216777    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-model-cache" (OuterVolumeSpecName: "model-cache") pod "73b55462-0f40-4b95-afa2-f1e71373cc38" (UID: "73b55462-0f40-4b95-afa2-f1e71373cc38"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:33:24.217014 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.216916    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:33:24.217296 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.217260    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b55462-0f40-4b95-afa2-f1e71373cc38-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "73b55462-0f40-4b95-afa2-f1e71373cc38" (UID: "73b55462-0f40-4b95-afa2-f1e71373cc38"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:33:24.217639 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.217615    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-home" (OuterVolumeSpecName: "home") pod "73b55462-0f40-4b95-afa2-f1e71373cc38" (UID: "73b55462-0f40-4b95-afa2-f1e71373cc38"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:33:24.218909 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.218875    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-dshm" (OuterVolumeSpecName: "dshm") pod "73b55462-0f40-4b95-afa2-f1e71373cc38" (UID: "73b55462-0f40-4b95-afa2-f1e71373cc38"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:33:24.219041 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.219024    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b55462-0f40-4b95-afa2-f1e71373cc38-kube-api-access-lmgxk" (OuterVolumeSpecName: "kube-api-access-lmgxk") pod "73b55462-0f40-4b95-afa2-f1e71373cc38" (UID: "73b55462-0f40-4b95-afa2-f1e71373cc38"). InnerVolumeSpecName "kube-api-access-lmgxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:33:24.219205 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.219182    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73b55462-0f40-4b95-afa2-f1e71373cc38" (UID: "73b55462-0f40-4b95-afa2-f1e71373cc38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:33:24.230032 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.230006    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "73b55462-0f40-4b95-afa2-f1e71373cc38" (UID: "73b55462-0f40-4b95-afa2-f1e71373cc38"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:33:24.240809 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.240783    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "73b55462-0f40-4b95-afa2-f1e71373cc38" (UID: "73b55462-0f40-4b95-afa2-f1e71373cc38"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:33:24.317307 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.317269    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:33:24.317307 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.317303    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:33:24.317535 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.317315    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73b55462-0f40-4b95-afa2-f1e71373cc38-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:33:24.317535 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.317326    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:33:24.317535 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.317335    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73b55462-0f40-4b95-afa2-f1e71373cc38-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:33:24.317535 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.317344    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b55462-0f40-4b95-afa2-f1e71373cc38-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:33:24.317535 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.317373    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmgxk\" (UniqueName: \"kubernetes.io/projected/73b55462-0f40-4b95-afa2-f1e71373cc38-kube-api-access-lmgxk\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:33:24.401994 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.401954    2578 generic.go:356] "Generic (PLEG): container finished" podID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerID="b85d2a62f4369071f9bb190fe5ddec803d56a549489445d428062903cbd783e8" exitCode=0
Jul 31 15:33:24.402156 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.402005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" event={"ID":"73b55462-0f40-4b95-afa2-f1e71373cc38","Type":"ContainerDied","Data":"b85d2a62f4369071f9bb190fe5ddec803d56a549489445d428062903cbd783e8"}
Jul 31 15:33:24.402156 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.402011    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"
Jul 31 15:33:24.402156 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.402090    2578 scope.go:117] "RemoveContainer" containerID="b85d2a62f4369071f9bb190fe5ddec803d56a549489445d428062903cbd783e8"
Jul 31 15:33:24.402278 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.402076    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv" event={"ID":"73b55462-0f40-4b95-afa2-f1e71373cc38","Type":"ContainerDied","Data":"59e11d5000d8483e19ae3fdbd7edc8decef8bc55cca073120e01c1591ebec8e0"}
Jul 31 15:33:24.410868 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.410851    2578 scope.go:117] "RemoveContainer" containerID="8556cc11d078c615aca078d097a38e780ff78c0968df6b746322bd118f6e6da2"
Jul 31 15:33:24.418120 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.418102    2578 scope.go:117] "RemoveContainer" containerID="b85d2a62f4369071f9bb190fe5ddec803d56a549489445d428062903cbd783e8"
Jul 31 15:33:24.418381 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:33:24.418362    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b85d2a62f4369071f9bb190fe5ddec803d56a549489445d428062903cbd783e8\": container with ID starting with b85d2a62f4369071f9bb190fe5ddec803d56a549489445d428062903cbd783e8 not found: ID does not exist" containerID="b85d2a62f4369071f9bb190fe5ddec803d56a549489445d428062903cbd783e8"
Jul 31 15:33:24.418433 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.418390    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85d2a62f4369071f9bb190fe5ddec803d56a549489445d428062903cbd783e8"} err="failed to get container status \"b85d2a62f4369071f9bb190fe5ddec803d56a549489445d428062903cbd783e8\": rpc error: code = NotFound desc = could not find container \"b85d2a62f4369071f9bb190fe5ddec803d56a549489445d428062903cbd783e8\": container with ID starting with b85d2a62f4369071f9bb190fe5ddec803d56a549489445d428062903cbd783e8 not found: ID does not exist"
Jul 31 15:33:24.418433 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.418410    2578 scope.go:117] "RemoveContainer" containerID="8556cc11d078c615aca078d097a38e780ff78c0968df6b746322bd118f6e6da2"
Jul 31 15:33:24.418654 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:33:24.418638    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8556cc11d078c615aca078d097a38e780ff78c0968df6b746322bd118f6e6da2\": container with ID starting with 8556cc11d078c615aca078d097a38e780ff78c0968df6b746322bd118f6e6da2 not found: ID does not exist" containerID="8556cc11d078c615aca078d097a38e780ff78c0968df6b746322bd118f6e6da2"
Jul 31 15:33:24.418704 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.418656    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8556cc11d078c615aca078d097a38e780ff78c0968df6b746322bd118f6e6da2"} err="failed to get container status \"8556cc11d078c615aca078d097a38e780ff78c0968df6b746322bd118f6e6da2\": rpc error: code = NotFound desc = could not find container \"8556cc11d078c615aca078d097a38e780ff78c0968df6b746322bd118f6e6da2\": container with ID starting with 8556cc11d078c615aca078d097a38e780ff78c0968df6b746322bd118f6e6da2 not found: ID does not exist"
Jul 31 15:33:24.425127 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.425093    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"]
Jul 31 15:33:24.429492 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.429463    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-jrnfv"]
Jul 31 15:33:24.489659 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:24.489628    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" path="/var/lib/kubelet/pods/73b55462-0f40-4b95-afa2-f1e71373cc38/volumes"
Jul 31 15:33:28.517177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.517145    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"]
Jul 31 15:33:28.517554 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.517415    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="886c4367-2a51-4431-a847-4cafe8458840" containerName="istio-proxy"
Jul 31 15:33:28.517554 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.517426    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="886c4367-2a51-4431-a847-4cafe8458840" containerName="istio-proxy"
Jul 31 15:33:28.517554 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.517438    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="886c4367-2a51-4431-a847-4cafe8458840" containerName="istio-proxy"
Jul 31 15:33:28.517554 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.517445    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="886c4367-2a51-4431-a847-4cafe8458840" containerName="istio-proxy"
Jul 31 15:33:28.517554 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.517454    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerName="storage-initializer"
Jul 31 15:33:28.517554 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.517460    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerName="storage-initializer"
Jul 31 15:33:28.517554 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.517466    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerName="main"
Jul 31 15:33:28.517554 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.517472    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerName="main"
Jul 31 15:33:28.517554 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.517525    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="886c4367-2a51-4431-a847-4cafe8458840" containerName="istio-proxy"
Jul 31 15:33:28.517554 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.517532    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="73b55462-0f40-4b95-afa2-f1e71373cc38" containerName="main"
Jul 31 15:33:28.517865 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.517620    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="886c4367-2a51-4431-a847-4cafe8458840" containerName="istio-proxy"
Jul 31 15:33:28.520289 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.520271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.522956 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.522930    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 31 15:33:28.523471 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.523450    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 31 15:33:28.523601 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.523459    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 31 15:33:28.523601 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.523508    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-xpppn\""
Jul 31 15:33:28.523601 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.523525    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:33:28.523601 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.523463    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 31 15:33:28.533076 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.533054    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"]
Jul 31 15:33:28.549791 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.549763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ksjr\" (UniqueName: \"kubernetes.io/projected/d8b19807-16aa-4048-804c-bd833f4174a7-kube-api-access-5ksjr\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.549894 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.549798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-model-cache\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.549894 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.549819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-tmp-dir\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.549894 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.549845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-dshm\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.550022 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.549892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-home\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.550022 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.549913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8b19807-16aa-4048-804c-bd833f4174a7-cabundle-cert\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.550022 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.549932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8b19807-16aa-4048-804c-bd833f4174a7-tls-certs\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.550022 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.549956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-kserve-provision-location\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.651024 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.650983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-kserve-provision-location\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.651192 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.651031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ksjr\" (UniqueName: \"kubernetes.io/projected/d8b19807-16aa-4048-804c-bd833f4174a7-kube-api-access-5ksjr\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.651192 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.651058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-model-cache\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.651192 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.651084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-tmp-dir\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.651192 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.651158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-dshm\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.651442 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.651204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-home\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.651442 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.651231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8b19807-16aa-4048-804c-bd833f4174a7-cabundle-cert\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.651442 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.651264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8b19807-16aa-4048-804c-bd833f4174a7-tls-certs\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.651602 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.651442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-kserve-provision-location\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.651602 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.651490    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-tmp-dir\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.651602 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.651561    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-model-cache\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.651708 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.651627    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-home\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.651970 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.651952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8b19807-16aa-4048-804c-bd833f4174a7-cabundle-cert\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.653572 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.653550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-dshm\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.653836 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.653820    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8b19807-16aa-4048-804c-bd833f4174a7-tls-certs\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.659621 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.659572    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ksjr\" (UniqueName: \"kubernetes.io/projected/d8b19807-16aa-4048-804c-bd833f4174a7-kube-api-access-5ksjr\") pod \"stop-feature-test-kserve-7648cf7688-cfrnj\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.701228 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.701192    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"]
Jul 31 15:33:28.704926 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.704904    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"
Jul 31 15:33:28.708748 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.708727    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-rgz8q\""
Jul 31 15:33:28.714205 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.714183    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"]
Jul 31 15:33:28.752447 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.752418    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43817a9-3fe7-4c14-ac36-046312bbbf8d-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8\" (UID: \"f43817a9-3fe7-4c14-ac36-046312bbbf8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"
Jul 31 15:33:28.752602 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.752453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqhwg\" (UniqueName: \"kubernetes.io/projected/f43817a9-3fe7-4c14-ac36-046312bbbf8d-kube-api-access-tqhwg\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8\" (UID: \"f43817a9-3fe7-4c14-ac36-046312bbbf8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"
Jul 31 15:33:28.831151 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.831072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:28.853593 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.853560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43817a9-3fe7-4c14-ac36-046312bbbf8d-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8\" (UID: \"f43817a9-3fe7-4c14-ac36-046312bbbf8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"
Jul 31 15:33:28.853694 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.853597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqhwg\" (UniqueName: \"kubernetes.io/projected/f43817a9-3fe7-4c14-ac36-046312bbbf8d-kube-api-access-tqhwg\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8\" (UID: \"f43817a9-3fe7-4c14-ac36-046312bbbf8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"
Jul 31 15:33:28.856065 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.856035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43817a9-3fe7-4c14-ac36-046312bbbf8d-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8\" (UID: \"f43817a9-3fe7-4c14-ac36-046312bbbf8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"
Jul 31 15:33:28.863187 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.863162    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqhwg\" (UniqueName: \"kubernetes.io/projected/f43817a9-3fe7-4c14-ac36-046312bbbf8d-kube-api-access-tqhwg\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8\" (UID: \"f43817a9-3fe7-4c14-ac36-046312bbbf8d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"
Jul 31 15:33:28.966505 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:28.966478    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"]
Jul 31 15:33:28.968968 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:33:28.968941    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b19807_16aa_4048_804c_bd833f4174a7.slice/crio-d5cfbb877dccdc2eb1a225d3d130806c3257ddc1b53fa6dc29565911d7dc30d8 WatchSource:0}: Error finding container d5cfbb877dccdc2eb1a225d3d130806c3257ddc1b53fa6dc29565911d7dc30d8: Status 404 returned error can't find the container with id d5cfbb877dccdc2eb1a225d3d130806c3257ddc1b53fa6dc29565911d7dc30d8
Jul 31 15:33:29.015926 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:29.015906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"
Jul 31 15:33:29.150263 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:29.150209    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"]
Jul 31 15:33:29.153264 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:33:29.153230    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43817a9_3fe7_4c14_ac36_046312bbbf8d.slice/crio-e5cd2c48a98154dc20f5c2d1766dc0721361c8c15bcc33c49254d18758946ac9 WatchSource:0}: Error finding container e5cd2c48a98154dc20f5c2d1766dc0721361c8c15bcc33c49254d18758946ac9: Status 404 returned error can't find the container with id e5cd2c48a98154dc20f5c2d1766dc0721361c8c15bcc33c49254d18758946ac9
Jul 31 15:33:29.424899 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:29.424802    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8" event={"ID":"f43817a9-3fe7-4c14-ac36-046312bbbf8d","Type":"ContainerStarted","Data":"9254a4cf1de06bdf2789c9090e8fc79704963c8d73c5c1343636d07da0e39869"}
Jul 31 15:33:29.425126 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:29.425100    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8" event={"ID":"f43817a9-3fe7-4c14-ac36-046312bbbf8d","Type":"ContainerStarted","Data":"e5cd2c48a98154dc20f5c2d1766dc0721361c8c15bcc33c49254d18758946ac9"}
Jul 31 15:33:29.425208 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:29.425145    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"
Jul 31 15:33:29.426645 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:29.426557    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" event={"ID":"d8b19807-16aa-4048-804c-bd833f4174a7","Type":"ContainerStarted","Data":"e4d6bf0735278cc6c89cd00d1e20ed802925def256d2de67ca103e4fbe537ae7"}
Jul 31 15:33:29.426645 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:29.426612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" event={"ID":"d8b19807-16aa-4048-804c-bd833f4174a7","Type":"ContainerStarted","Data":"d5cfbb877dccdc2eb1a225d3d130806c3257ddc1b53fa6dc29565911d7dc30d8"}
Jul 31 15:33:29.469407 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:29.469318    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8" podStartSLOduration=1.469297415 podStartE2EDuration="1.469297415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:33:29.446280369 +0000 UTC m=+2707.455492238" watchObservedRunningTime="2026-07-31 15:33:29.469297415 +0000 UTC m=+2707.478509286"
Jul 31 15:33:31.434934 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:31.434896    2578 generic.go:356] "Generic (PLEG): container finished" podID="d8b19807-16aa-4048-804c-bd833f4174a7" containerID="e4d6bf0735278cc6c89cd00d1e20ed802925def256d2de67ca103e4fbe537ae7" exitCode=0
Jul 31 15:33:31.435450 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:31.434956    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" event={"ID":"d8b19807-16aa-4048-804c-bd833f4174a7","Type":"ContainerDied","Data":"e4d6bf0735278cc6c89cd00d1e20ed802925def256d2de67ca103e4fbe537ae7"}
Jul 31 15:33:32.440114 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:32.440078    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" event={"ID":"d8b19807-16aa-4048-804c-bd833f4174a7","Type":"ContainerStarted","Data":"4e1165d0bc403df5efed8476eef5c97772cd3ecc03788b937958e35b71e2c4de"}
Jul 31 15:33:32.466508 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:32.466441    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" podStartSLOduration=2.277027052 podStartE2EDuration="4.466419833s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.189392781s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:33:32.465055672 +0000 UTC m=+2710.474267543" watchObservedRunningTime="2026-07-31 15:33:32.466419833 +0000 UTC m=+2710.475631703"
Jul 31 15:33:32.832217 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:32.832170    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:38.831677 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:38.831635    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:33:38.833213 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:38.833184    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 31 15:33:48.831898 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:48.831854    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 31 15:33:58.831814 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:33:58.831765    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 31 15:34:00.432629 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:34:00.432595    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"
Jul 31 15:34:08.832593 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:34:08.832544    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 31 15:34:18.832531 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:34:18.832485    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 31 15:34:28.831894 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:34:28.831847    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 31 15:34:38.831907 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:34:38.831852    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 31 15:34:48.832251 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:34:48.832149    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 31 15:34:58.831736 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:34:58.831678    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 31 15:35:08.832071 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:08.832030    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 31 15:35:18.832527 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:18.832476    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 31 15:35:28.842045 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:28.842013    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:35:28.850172 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:28.850149    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:35:29.906323 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:29.906287    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"]
Jul 31 15:35:29.910990 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:29.910960    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"]
Jul 31 15:35:30.432924 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:30.432884    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8" podUID="f43817a9-3fe7-4c14-ac36-046312bbbf8d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 15:35:40.432803 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:40.432763    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8" podUID="f43817a9-3fe7-4c14-ac36-046312bbbf8d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 15:35:44.929829 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:44.929775    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8" podUID="f43817a9-3fe7-4c14-ac36-046312bbbf8d" containerName="main" containerID="cri-o://9254a4cf1de06bdf2789c9090e8fc79704963c8d73c5c1343636d07da0e39869" gracePeriod=45
Jul 31 15:35:44.943040 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:44.943002    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" containerName="main" containerID="cri-o://4e1165d0bc403df5efed8476eef5c97772cd3ecc03788b937958e35b71e2c4de" gracePeriod=45
Jul 31 15:35:45.171683 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.171653    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"
Jul 31 15:35:45.252795 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.252756    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqhwg\" (UniqueName: \"kubernetes.io/projected/f43817a9-3fe7-4c14-ac36-046312bbbf8d-kube-api-access-tqhwg\") pod \"f43817a9-3fe7-4c14-ac36-046312bbbf8d\" (UID: \"f43817a9-3fe7-4c14-ac36-046312bbbf8d\") "
Jul 31 15:35:45.253001 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.252819    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43817a9-3fe7-4c14-ac36-046312bbbf8d-tls-certs\") pod \"f43817a9-3fe7-4c14-ac36-046312bbbf8d\" (UID: \"f43817a9-3fe7-4c14-ac36-046312bbbf8d\") "
Jul 31 15:35:45.254835 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.254797    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f43817a9-3fe7-4c14-ac36-046312bbbf8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f43817a9-3fe7-4c14-ac36-046312bbbf8d" (UID: "f43817a9-3fe7-4c14-ac36-046312bbbf8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:35:45.254943 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.254909    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43817a9-3fe7-4c14-ac36-046312bbbf8d-kube-api-access-tqhwg" (OuterVolumeSpecName: "kube-api-access-tqhwg") pod "f43817a9-3fe7-4c14-ac36-046312bbbf8d" (UID: "f43817a9-3fe7-4c14-ac36-046312bbbf8d"). InnerVolumeSpecName "kube-api-access-tqhwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:35:45.353663 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.353626    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqhwg\" (UniqueName: \"kubernetes.io/projected/f43817a9-3fe7-4c14-ac36-046312bbbf8d-kube-api-access-tqhwg\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:35:45.353831 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.353673    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43817a9-3fe7-4c14-ac36-046312bbbf8d-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:35:45.677877 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.677852    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:35:45.757402 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.757338    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-dshm\") pod \"d8b19807-16aa-4048-804c-bd833f4174a7\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") "
Jul 31 15:35:45.757586 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.757417    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-model-cache\") pod \"d8b19807-16aa-4048-804c-bd833f4174a7\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") "
Jul 31 15:35:45.757586 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.757446    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ksjr\" (UniqueName: \"kubernetes.io/projected/d8b19807-16aa-4048-804c-bd833f4174a7-kube-api-access-5ksjr\") pod \"d8b19807-16aa-4048-804c-bd833f4174a7\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") "
Jul 31 15:35:45.757586 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.757468    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-tmp-dir\") pod \"d8b19807-16aa-4048-804c-bd833f4174a7\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") "
Jul 31 15:35:45.757586 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.757499    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8b19807-16aa-4048-804c-bd833f4174a7-tls-certs\") pod \"d8b19807-16aa-4048-804c-bd833f4174a7\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") "
Jul 31 15:35:45.757818 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.757673    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-kserve-provision-location\") pod \"d8b19807-16aa-4048-804c-bd833f4174a7\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") "
Jul 31 15:35:45.757818 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.757721    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-home\") pod \"d8b19807-16aa-4048-804c-bd833f4174a7\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") "
Jul 31 15:35:45.757818 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.757750    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8b19807-16aa-4048-804c-bd833f4174a7" (UID: "d8b19807-16aa-4048-804c-bd833f4174a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:35:45.757818 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.757786    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8b19807-16aa-4048-804c-bd833f4174a7-cabundle-cert\") pod \"d8b19807-16aa-4048-804c-bd833f4174a7\" (UID: \"d8b19807-16aa-4048-804c-bd833f4174a7\") "
Jul 31 15:35:45.758043 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.758023    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:35:45.758488 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.758457    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b19807-16aa-4048-804c-bd833f4174a7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d8b19807-16aa-4048-804c-bd833f4174a7" (UID: "d8b19807-16aa-4048-804c-bd833f4174a7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:35:45.758613 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.758522    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-home" (OuterVolumeSpecName: "home") pod "d8b19807-16aa-4048-804c-bd833f4174a7" (UID: "d8b19807-16aa-4048-804c-bd833f4174a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:35:45.759608 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.759584    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-dshm" (OuterVolumeSpecName: "dshm") pod "d8b19807-16aa-4048-804c-bd833f4174a7" (UID: "d8b19807-16aa-4048-804c-bd833f4174a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:35:45.759705 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.759647    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8b19807-16aa-4048-804c-bd833f4174a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8b19807-16aa-4048-804c-bd833f4174a7" (UID: "d8b19807-16aa-4048-804c-bd833f4174a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:35:45.759768 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.759748    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b19807-16aa-4048-804c-bd833f4174a7-kube-api-access-5ksjr" (OuterVolumeSpecName: "kube-api-access-5ksjr") pod "d8b19807-16aa-4048-804c-bd833f4174a7" (UID: "d8b19807-16aa-4048-804c-bd833f4174a7"). InnerVolumeSpecName "kube-api-access-5ksjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:35:45.775970 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.775941    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8b19807-16aa-4048-804c-bd833f4174a7" (UID: "d8b19807-16aa-4048-804c-bd833f4174a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:35:45.787814 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.787789    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d8b19807-16aa-4048-804c-bd833f4174a7" (UID: "d8b19807-16aa-4048-804c-bd833f4174a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:35:45.859069 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.858996    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:35:45.859069 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.859027    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ksjr\" (UniqueName: \"kubernetes.io/projected/d8b19807-16aa-4048-804c-bd833f4174a7-kube-api-access-5ksjr\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:35:45.859069 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.859037    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:35:45.859069 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.859047    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8b19807-16aa-4048-804c-bd833f4174a7-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:35:45.859069 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.859055    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:35:45.859069 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.859063    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8b19807-16aa-4048-804c-bd833f4174a7-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:35:45.859069 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.859071    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8b19807-16aa-4048-804c-bd833f4174a7-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:35:45.981804 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.981765    2578 generic.go:356] "Generic (PLEG): container finished" podID="f43817a9-3fe7-4c14-ac36-046312bbbf8d" containerID="9254a4cf1de06bdf2789c9090e8fc79704963c8d73c5c1343636d07da0e39869" exitCode=0
Jul 31 15:35:45.982225 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.981814    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"
Jul 31 15:35:45.982225 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.981848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8" event={"ID":"f43817a9-3fe7-4c14-ac36-046312bbbf8d","Type":"ContainerDied","Data":"9254a4cf1de06bdf2789c9090e8fc79704963c8d73c5c1343636d07da0e39869"}
Jul 31 15:35:45.982225 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.981902    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8" event={"ID":"f43817a9-3fe7-4c14-ac36-046312bbbf8d","Type":"ContainerDied","Data":"e5cd2c48a98154dc20f5c2d1766dc0721361c8c15bcc33c49254d18758946ac9"}
Jul 31 15:35:45.982225 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.981925    2578 scope.go:117] "RemoveContainer" containerID="9254a4cf1de06bdf2789c9090e8fc79704963c8d73c5c1343636d07da0e39869"
Jul 31 15:35:45.983731 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.983710    2578 generic.go:356] "Generic (PLEG): container finished" podID="d8b19807-16aa-4048-804c-bd833f4174a7" containerID="4e1165d0bc403df5efed8476eef5c97772cd3ecc03788b937958e35b71e2c4de" exitCode=0
Jul 31 15:35:45.983853 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.983753    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" event={"ID":"d8b19807-16aa-4048-804c-bd833f4174a7","Type":"ContainerDied","Data":"4e1165d0bc403df5efed8476eef5c97772cd3ecc03788b937958e35b71e2c4de"}
Jul 31 15:35:45.983853 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.983769    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"
Jul 31 15:35:45.983853 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.983787    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj" event={"ID":"d8b19807-16aa-4048-804c-bd833f4174a7","Type":"ContainerDied","Data":"d5cfbb877dccdc2eb1a225d3d130806c3257ddc1b53fa6dc29565911d7dc30d8"}
Jul 31 15:35:45.990456 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.990429    2578 scope.go:117] "RemoveContainer" containerID="9254a4cf1de06bdf2789c9090e8fc79704963c8d73c5c1343636d07da0e39869"
Jul 31 15:35:45.990715 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:35:45.990695    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9254a4cf1de06bdf2789c9090e8fc79704963c8d73c5c1343636d07da0e39869\": container with ID starting with 9254a4cf1de06bdf2789c9090e8fc79704963c8d73c5c1343636d07da0e39869 not found: ID does not exist" containerID="9254a4cf1de06bdf2789c9090e8fc79704963c8d73c5c1343636d07da0e39869"
Jul 31 15:35:45.990793 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.990721    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9254a4cf1de06bdf2789c9090e8fc79704963c8d73c5c1343636d07da0e39869"} err="failed to get container status \"9254a4cf1de06bdf2789c9090e8fc79704963c8d73c5c1343636d07da0e39869\": rpc error: code = NotFound desc = could not find container \"9254a4cf1de06bdf2789c9090e8fc79704963c8d73c5c1343636d07da0e39869\": container with ID starting with 9254a4cf1de06bdf2789c9090e8fc79704963c8d73c5c1343636d07da0e39869 not found: ID does not exist"
Jul 31 15:35:45.990793 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.990738    2578 scope.go:117] "RemoveContainer" containerID="4e1165d0bc403df5efed8476eef5c97772cd3ecc03788b937958e35b71e2c4de"
Jul 31 15:35:45.998278 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:45.998262    2578 scope.go:117] "RemoveContainer" containerID="e4d6bf0735278cc6c89cd00d1e20ed802925def256d2de67ca103e4fbe537ae7"
Jul 31 15:35:46.005465 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:46.005441    2578 scope.go:117] "RemoveContainer" containerID="4e1165d0bc403df5efed8476eef5c97772cd3ecc03788b937958e35b71e2c4de"
Jul 31 15:35:46.005678 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:35:46.005661    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1165d0bc403df5efed8476eef5c97772cd3ecc03788b937958e35b71e2c4de\": container with ID starting with 4e1165d0bc403df5efed8476eef5c97772cd3ecc03788b937958e35b71e2c4de not found: ID does not exist" containerID="4e1165d0bc403df5efed8476eef5c97772cd3ecc03788b937958e35b71e2c4de"
Jul 31 15:35:46.005723 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:46.005685    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1165d0bc403df5efed8476eef5c97772cd3ecc03788b937958e35b71e2c4de"} err="failed to get container status \"4e1165d0bc403df5efed8476eef5c97772cd3ecc03788b937958e35b71e2c4de\": rpc error: code = NotFound desc = could not find container \"4e1165d0bc403df5efed8476eef5c97772cd3ecc03788b937958e35b71e2c4de\": container with ID starting with 4e1165d0bc403df5efed8476eef5c97772cd3ecc03788b937958e35b71e2c4de not found: ID does not exist"
Jul 31 15:35:46.005723 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:46.005700    2578 scope.go:117] "RemoveContainer" containerID="e4d6bf0735278cc6c89cd00d1e20ed802925def256d2de67ca103e4fbe537ae7"
Jul 31 15:35:46.005926 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:35:46.005908    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d6bf0735278cc6c89cd00d1e20ed802925def256d2de67ca103e4fbe537ae7\": container with ID starting with e4d6bf0735278cc6c89cd00d1e20ed802925def256d2de67ca103e4fbe537ae7 not found: ID does not exist" containerID="e4d6bf0735278cc6c89cd00d1e20ed802925def256d2de67ca103e4fbe537ae7"
Jul 31 15:35:46.005975 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:46.005931    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d6bf0735278cc6c89cd00d1e20ed802925def256d2de67ca103e4fbe537ae7"} err="failed to get container status \"e4d6bf0735278cc6c89cd00d1e20ed802925def256d2de67ca103e4fbe537ae7\": rpc error: code = NotFound desc = could not find container \"e4d6bf0735278cc6c89cd00d1e20ed802925def256d2de67ca103e4fbe537ae7\": container with ID starting with e4d6bf0735278cc6c89cd00d1e20ed802925def256d2de67ca103e4fbe537ae7 not found: ID does not exist"
Jul 31 15:35:46.012184 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:46.012164    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"]
Jul 31 15:35:46.016901 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:46.016880    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7648cf7688-cfrnj"]
Jul 31 15:35:46.028062 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:46.028038    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"]
Jul 31 15:35:46.030294 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:46.030263    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-h79d8"]
Jul 31 15:35:46.489484 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:46.489451    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" path="/var/lib/kubelet/pods/d8b19807-16aa-4048-804c-bd833f4174a7/volumes"
Jul 31 15:35:46.489851 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:35:46.489838    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f43817a9-3fe7-4c14-ac36-046312bbbf8d" path="/var/lib/kubelet/pods/f43817a9-3fe7-4c14-ac36-046312bbbf8d/volumes"
Jul 31 15:36:08.194769 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.194733    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"]
Jul 31 15:36:08.195202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.194998    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f43817a9-3fe7-4c14-ac36-046312bbbf8d" containerName="main"
Jul 31 15:36:08.195202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.195011    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f43817a9-3fe7-4c14-ac36-046312bbbf8d" containerName="main"
Jul 31 15:36:08.195202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.195027    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" containerName="storage-initializer"
Jul 31 15:36:08.195202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.195033    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" containerName="storage-initializer"
Jul 31 15:36:08.195202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.195040    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" containerName="main"
Jul 31 15:36:08.195202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.195046    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" containerName="main"
Jul 31 15:36:08.195202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.195095    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b19807-16aa-4048-804c-bd833f4174a7" containerName="main"
Jul 31 15:36:08.195202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.195103    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f43817a9-3fe7-4c14-ac36-046312bbbf8d" containerName="main"
Jul 31 15:36:08.197118 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.197100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.199841 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.199810    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Jul 31 15:36:08.199987 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.199876    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Jul 31 15:36:08.200398 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.200375    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-wstx5\""
Jul 31 15:36:08.200694 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.200381    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Jul 31 15:36:08.200829 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.200389    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 31 15:36:08.200829 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.200445    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:36:08.210223 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.210203    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"]
Jul 31 15:36:08.227813 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.227787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-model-cache\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.227933 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.227819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.227933 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.227839    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-home\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.227933 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.227871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mql5\" (UniqueName: \"kubernetes.io/projected/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-kube-api-access-4mql5\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.227933 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.227909    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-dshm\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.227933 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.227926    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-cabundle-cert\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.228110 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.228013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-tmp-dir\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.228110 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.228063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-tls-certs\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.328634 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.328594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-tmp-dir\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.328634 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.328640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-tls-certs\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.328872 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.328679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-model-cache\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.328872 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.328699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.328872 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.328723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-home\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.328872 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.328759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mql5\" (UniqueName: \"kubernetes.io/projected/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-kube-api-access-4mql5\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.328872 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.328789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-dshm\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.329202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.329045    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-tmp-dir\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.329202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.329090    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-model-cache\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.329202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.329110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-cabundle-cert\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.329202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.329134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.329427 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.329249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-home\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.329661 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.329642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-cabundle-cert\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.331097 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.331071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-dshm\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.331394 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.331378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-tls-certs\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.337435 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.337412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mql5\" (UniqueName: \"kubernetes.io/projected/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-kube-api-access-4mql5\") pod \"lora-single-adapter-test-kserve-84c944695d-8jqdv\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.508057 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.508021    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:08.650533 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:08.649539    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"]
Jul 31 15:36:08.652249 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:36:08.652215    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b87d69e_dca0_4b7e_afc7_2bd42ae2c8d0.slice/crio-204928e2e530ee31f3895b97e69ec59e70658c2360253381fbf5889133f4ebf6 WatchSource:0}: Error finding container 204928e2e530ee31f3895b97e69ec59e70658c2360253381fbf5889133f4ebf6: Status 404 returned error can't find the container with id 204928e2e530ee31f3895b97e69ec59e70658c2360253381fbf5889133f4ebf6
Jul 31 15:36:09.071218 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:09.071173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv" event={"ID":"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0","Type":"ContainerStarted","Data":"c1c725e2a1033b6a6c569e0dac4dda6a87f81819ac46e3186395cf646555778c"}
Jul 31 15:36:09.071409 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:09.071279    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv" event={"ID":"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0","Type":"ContainerStarted","Data":"204928e2e530ee31f3895b97e69ec59e70658c2360253381fbf5889133f4ebf6"}
Jul 31 15:36:12.083877 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:12.083840    2578 generic.go:356] "Generic (PLEG): container finished" podID="9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" containerID="c1c725e2a1033b6a6c569e0dac4dda6a87f81819ac46e3186395cf646555778c" exitCode=0
Jul 31 15:36:12.084392 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:12.083911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv" event={"ID":"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0","Type":"ContainerDied","Data":"c1c725e2a1033b6a6c569e0dac4dda6a87f81819ac46e3186395cf646555778c"}
Jul 31 15:36:13.090066 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:13.090025    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv" event={"ID":"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0","Type":"ContainerStarted","Data":"8ac3c865701bcca199281780e431e5de1d931b97aa1e81524ab3758c76a1dabe"}
Jul 31 15:36:13.113316 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:13.113268    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv" podStartSLOduration=2.115795195 podStartE2EDuration="5.113254014s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.997458819s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:36:13.11138356 +0000 UTC m=+2871.120595431" watchObservedRunningTime="2026-07-31 15:36:13.113254014 +0000 UTC m=+2871.122465884"
Jul 31 15:36:13.509022 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:13.508977    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:18.508433 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:18.508328    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:36:18.509940 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:18.509909    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv" podUID="9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 31 15:36:28.508887 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:28.508840    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv" podUID="9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 31 15:36:38.508964 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:38.508921    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv" podUID="9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 31 15:36:48.508641 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:48.508591    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv" podUID="9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 31 15:36:58.509481 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:36:58.509439    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv" podUID="9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 31 15:37:03.640224 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.640186    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"]
Jul 31 15:37:03.643875 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.643846    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.646838 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.646811    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Jul 31 15:37:03.647095 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.646828    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-l8m6t\""
Jul 31 15:37:03.647095 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.646870    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Jul 31 15:37:03.647264 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.646916    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Jul 31 15:37:03.655706 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.655673    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"]
Jul 31 15:37:03.690610 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.690565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-istio-envoy\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.690812 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.690618    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-istio-data\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.690812 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.690641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975ce940-485d-46cf-8760-b35155f58567-istio-podinfo\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.690812 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.690664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975ce940-485d-46cf-8760-b35155f58567-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.690812 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.690681    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s4vn\" (UniqueName: \"kubernetes.io/projected/975ce940-485d-46cf-8760-b35155f58567-kube-api-access-8s4vn\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.690812 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.690726    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975ce940-485d-46cf-8760-b35155f58567-istio-token\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.690812 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.690793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-workload-socket\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.691045 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.690824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-credential-socket\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.691045 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.690840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-workload-certs\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.791950 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.791842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-workload-certs\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.791950 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.791908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-istio-envoy\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.791950 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.791942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-istio-data\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.792406 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.791968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975ce940-485d-46cf-8760-b35155f58567-istio-podinfo\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.792406 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.791998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975ce940-485d-46cf-8760-b35155f58567-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.792406 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.792035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s4vn\" (UniqueName: \"kubernetes.io/projected/975ce940-485d-46cf-8760-b35155f58567-kube-api-access-8s4vn\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.792406 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.792058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975ce940-485d-46cf-8760-b35155f58567-istio-token\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.792406 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.792145    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-workload-socket\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.792406 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.792189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-credential-socket\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.792406 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.792339    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-workload-certs\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.792834 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.792322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-istio-data\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.792834 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.792562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-credential-socket\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.792834 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.792623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-workload-socket\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.793174 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.793152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975ce940-485d-46cf-8760-b35155f58567-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.794359 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.794321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-istio-envoy\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.794775 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.794756    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975ce940-485d-46cf-8760-b35155f58567-istio-podinfo\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.801883 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.801857    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975ce940-485d-46cf-8760-b35155f58567-istio-token\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.802070 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.802047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s4vn\" (UniqueName: \"kubernetes.io/projected/975ce940-485d-46cf-8760-b35155f58567-kube-api-access-8s4vn\") pod \"router-gateway-1-openshift-default-8675cdb9d8-f7lgd\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:03.956811 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:03.956766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:04.097568 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:04.097521    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"]
Jul 31 15:37:04.100480 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:37:04.100449    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975ce940_485d_46cf_8760_b35155f58567.slice/crio-b91f6b8c011233040d255cb36a6246f3b731bf94476f80cc8c8b1298021492ae WatchSource:0}: Error finding container b91f6b8c011233040d255cb36a6246f3b731bf94476f80cc8c8b1298021492ae: Status 404 returned error can't find the container with id b91f6b8c011233040d255cb36a6246f3b731bf94476f80cc8c8b1298021492ae
Jul 31 15:37:04.102705 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:04.102667    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:37:04.102817 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:04.102747    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:37:04.102817 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:04.102800    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:37:04.297768 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:04.297663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd" event={"ID":"975ce940-485d-46cf-8760-b35155f58567","Type":"ContainerStarted","Data":"f935c6a2b542376afe82ca819666a6865986e5cbcbfe54ec656067e8f28d7bfd"}
Jul 31 15:37:04.297768 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:04.297737    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd" event={"ID":"975ce940-485d-46cf-8760-b35155f58567","Type":"ContainerStarted","Data":"b91f6b8c011233040d255cb36a6246f3b731bf94476f80cc8c8b1298021492ae"}
Jul 31 15:37:04.323404 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:04.323324    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd" podStartSLOduration=1.3233056730000001 podStartE2EDuration="1.323305673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:37:04.322340663 +0000 UTC m=+2922.331552534" watchObservedRunningTime="2026-07-31 15:37:04.323305673 +0000 UTC m=+2922.332517536"
Jul 31 15:37:04.957169 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:04.957127    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:05.962092 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:05.962060    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:06.305886 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:06.305852    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:06.306779 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:06.306753    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:08.508555 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:08.508514    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv" podUID="9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 31 15:37:15.344942 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:15.344901    2578 generic.go:356] "Generic (PLEG): container finished" podID="975ce940-485d-46cf-8760-b35155f58567" containerID="f935c6a2b542376afe82ca819666a6865986e5cbcbfe54ec656067e8f28d7bfd" exitCode=0
Jul 31 15:37:15.345501 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:15.344953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd" event={"ID":"975ce940-485d-46cf-8760-b35155f58567","Type":"ContainerDied","Data":"f935c6a2b542376afe82ca819666a6865986e5cbcbfe54ec656067e8f28d7bfd"}
Jul 31 15:37:15.345501 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:15.345294    2578 scope.go:117] "RemoveContainer" containerID="f935c6a2b542376afe82ca819666a6865986e5cbcbfe54ec656067e8f28d7bfd"
Jul 31 15:37:15.346231 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:15.346201    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:37:15.346313 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:15.346283    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:37:15.346406 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:15.346328    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:37:16.353729 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:16.353681    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd" event={"ID":"975ce940-485d-46cf-8760-b35155f58567","Type":"ContainerStarted","Data":"4c5605e0ef55713ba6ea3afe7be8a68d2b01a3b9b5207d05ab5c1870d4593fbf"}
Jul 31 15:37:16.957484 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:16.957438    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:16.958427 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:16.958389    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd" podUID="975ce940-485d-46cf-8760-b35155f58567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 31 15:37:17.960832 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:17.960778    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd" podUID="975ce940-485d-46cf-8760-b35155f58567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 31 15:37:18.508818 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:18.508772    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv" podUID="9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 31 15:37:18.957608 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:18.957510    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd" podUID="975ce940-485d-46cf-8760-b35155f58567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 31 15:37:19.957598 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:19.957541    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd" podUID="975ce940-485d-46cf-8760-b35155f58567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 31 15:37:20.957724 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:20.957661    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd" podUID="975ce940-485d-46cf-8760-b35155f58567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 31 15:37:21.306750 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:21.306707    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:21.961691 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:21.961657    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:21.962818 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:21.962794    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:37:28.518618 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:28.518584    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:37:28.526262 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:37:28.526230    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:38:15.665696 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:15.665654    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-84c944695d-8jqdv_9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0/storage-initializer/0.log"
Jul 31 15:38:15.674835 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:15.674789    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-84c944695d-8jqdv_9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0/main/0.log"
Jul 31 15:38:17.237980 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:17.237945    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"]
Jul 31 15:38:32.259717 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:32.259672    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv" podUID="9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" containerName="main" containerID="cri-o://8ac3c865701bcca199281780e431e5de1d931b97aa1e81524ab3758c76a1dabe" gracePeriod=45
Jul 31 15:38:32.520080 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:32.519984    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv" podUID="9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 31 15:38:32.902970 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:32.902942    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:38:33.059930 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.059837    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mql5\" (UniqueName: \"kubernetes.io/projected/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-kube-api-access-4mql5\") pod \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") "
Jul 31 15:38:33.059930 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.059878    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-home\") pod \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") "
Jul 31 15:38:33.059930 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.059914    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-tls-certs\") pod \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") "
Jul 31 15:38:33.060236 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.059938    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-kserve-provision-location\") pod \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") "
Jul 31 15:38:33.060236 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.059965    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-dshm\") pod \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") "
Jul 31 15:38:33.060236 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.060001    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-model-cache\") pod \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") "
Jul 31 15:38:33.060236 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.060046    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-cabundle-cert\") pod \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") "
Jul 31 15:38:33.060236 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.060070    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-tmp-dir\") pod \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\" (UID: \"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0\") "
Jul 31 15:38:33.060523 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.060251    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-home" (OuterVolumeSpecName: "home") pod "9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" (UID: "9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:38:33.060523 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.060295    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" (UID: "9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:38:33.060712 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.060674    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" (UID: "9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:38:33.062516 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.062341    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" (UID: "9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:38:33.062516 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.062473    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-kube-api-access-4mql5" (OuterVolumeSpecName: "kube-api-access-4mql5") pod "9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" (UID: "9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0"). InnerVolumeSpecName "kube-api-access-4mql5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:38:33.062769 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.062744    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-dshm" (OuterVolumeSpecName: "dshm") pod "9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" (UID: "9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:38:33.073724 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.073682    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" (UID: "9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:38:33.084584 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.084562    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" (UID: "9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:38:33.160667 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.160636    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:38:33.160667 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.160662    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:38:33.160900 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.160678    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:38:33.160900 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.160693    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:38:33.160900 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.160702    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:38:33.160900 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.160713    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mql5\" (UniqueName: \"kubernetes.io/projected/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-kube-api-access-4mql5\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:38:33.160900 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.160723    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:38:33.160900 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.160732    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:38:33.673492 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.673453    2578 generic.go:356] "Generic (PLEG): container finished" podID="9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" containerID="8ac3c865701bcca199281780e431e5de1d931b97aa1e81524ab3758c76a1dabe" exitCode=0
Jul 31 15:38:33.674017 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.673513    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"
Jul 31 15:38:33.674017 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.673549    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv" event={"ID":"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0","Type":"ContainerDied","Data":"8ac3c865701bcca199281780e431e5de1d931b97aa1e81524ab3758c76a1dabe"}
Jul 31 15:38:33.674017 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.673626    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv" event={"ID":"9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0","Type":"ContainerDied","Data":"204928e2e530ee31f3895b97e69ec59e70658c2360253381fbf5889133f4ebf6"}
Jul 31 15:38:33.674017 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.673664    2578 scope.go:117] "RemoveContainer" containerID="8ac3c865701bcca199281780e431e5de1d931b97aa1e81524ab3758c76a1dabe"
Jul 31 15:38:33.682187 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.682162    2578 scope.go:117] "RemoveContainer" containerID="c1c725e2a1033b6a6c569e0dac4dda6a87f81819ac46e3186395cf646555778c"
Jul 31 15:38:33.690047 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.690027    2578 scope.go:117] "RemoveContainer" containerID="8ac3c865701bcca199281780e431e5de1d931b97aa1e81524ab3758c76a1dabe"
Jul 31 15:38:33.690287 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:38:33.690268    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ac3c865701bcca199281780e431e5de1d931b97aa1e81524ab3758c76a1dabe\": container with ID starting with 8ac3c865701bcca199281780e431e5de1d931b97aa1e81524ab3758c76a1dabe not found: ID does not exist" containerID="8ac3c865701bcca199281780e431e5de1d931b97aa1e81524ab3758c76a1dabe"
Jul 31 15:38:33.690367 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.690295    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac3c865701bcca199281780e431e5de1d931b97aa1e81524ab3758c76a1dabe"} err="failed to get container status \"8ac3c865701bcca199281780e431e5de1d931b97aa1e81524ab3758c76a1dabe\": rpc error: code = NotFound desc = could not find container \"8ac3c865701bcca199281780e431e5de1d931b97aa1e81524ab3758c76a1dabe\": container with ID starting with 8ac3c865701bcca199281780e431e5de1d931b97aa1e81524ab3758c76a1dabe not found: ID does not exist"
Jul 31 15:38:33.690367 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.690319    2578 scope.go:117] "RemoveContainer" containerID="c1c725e2a1033b6a6c569e0dac4dda6a87f81819ac46e3186395cf646555778c"
Jul 31 15:38:33.690573 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:38:33.690556    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c725e2a1033b6a6c569e0dac4dda6a87f81819ac46e3186395cf646555778c\": container with ID starting with c1c725e2a1033b6a6c569e0dac4dda6a87f81819ac46e3186395cf646555778c not found: ID does not exist" containerID="c1c725e2a1033b6a6c569e0dac4dda6a87f81819ac46e3186395cf646555778c"
Jul 31 15:38:33.690623 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.690576    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c725e2a1033b6a6c569e0dac4dda6a87f81819ac46e3186395cf646555778c"} err="failed to get container status \"c1c725e2a1033b6a6c569e0dac4dda6a87f81819ac46e3186395cf646555778c\": rpc error: code = NotFound desc = could not find container \"c1c725e2a1033b6a6c569e0dac4dda6a87f81819ac46e3186395cf646555778c\": container with ID starting with c1c725e2a1033b6a6c569e0dac4dda6a87f81819ac46e3186395cf646555778c not found: ID does not exist"
Jul 31 15:38:33.704846 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.704820    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"]
Jul 31 15:38:33.708869 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:33.708846    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-84c944695d-8jqdv"]
Jul 31 15:38:34.490041 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:38:34.490004    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" path="/var/lib/kubelet/pods/9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0/volumes"
Jul 31 15:39:03.081391 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.081339    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"]
Jul 31 15:39:03.081783 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.081654    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" containerName="main"
Jul 31 15:39:03.081783 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.081668    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" containerName="main"
Jul 31 15:39:03.081783 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.081677    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" containerName="storage-initializer"
Jul 31 15:39:03.081783 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.081683    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" containerName="storage-initializer"
Jul 31 15:39:03.081783 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.081761    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b87d69e-dca0-4b7e-afc7-2bd42ae2c8d0" containerName="main"
Jul 31 15:39:03.086165 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.086148    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.089077 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.089052    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Jul 31 15:39:03.089615 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.089591    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 31 15:39:03.089736 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.089609    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Jul 31 15:39:03.089803 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.089767    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-2pq7z\""
Jul 31 15:39:03.090230 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.090212    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:39:03.090319 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.090229    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Jul 31 15:39:03.101673 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.101646    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"]
Jul 31 15:39:03.196077 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.196045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-home\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.196283 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.196078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-dshm\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.196283 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.196150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-model-cache\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.196283 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.196169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37165081-ee6e-488e-a49b-95fd82749cea-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.196283 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.196226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.196283 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.196272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37165081-ee6e-488e-a49b-95fd82749cea-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.196525 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.196306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.196525 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.196335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2msg\" (UniqueName: \"kubernetes.io/projected/37165081-ee6e-488e-a49b-95fd82749cea-kube-api-access-x2msg\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.297245 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.297204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2msg\" (UniqueName: \"kubernetes.io/projected/37165081-ee6e-488e-a49b-95fd82749cea-kube-api-access-x2msg\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.297464 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.297262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-home\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.297464 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.297281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-dshm\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.297464 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.297310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-model-cache\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.297464 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.297327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37165081-ee6e-488e-a49b-95fd82749cea-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.297464 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.297381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.297742 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.297502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37165081-ee6e-488e-a49b-95fd82749cea-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.297742 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.297534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.297742 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.297705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-home\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.297912 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.297750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-model-cache\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.298250 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.298223    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.298250 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.298243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37165081-ee6e-488e-a49b-95fd82749cea-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.298432 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.298243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.299770 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.299751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-dshm\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.300070 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.300050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37165081-ee6e-488e-a49b-95fd82749cea-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.307260 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.307233    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2msg\" (UniqueName: \"kubernetes.io/projected/37165081-ee6e-488e-a49b-95fd82749cea-kube-api-access-x2msg\") pod \"lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.397095 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.397008    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:03.563773 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.563736    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"]
Jul 31 15:39:03.564874 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:39:03.564843    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37165081_ee6e_488e_a49b_95fd82749cea.slice/crio-5679037b34cb709b56cc868eac39f2949fab0e99805b38b15e40b9ba39a05980 WatchSource:0}: Error finding container 5679037b34cb709b56cc868eac39f2949fab0e99805b38b15e40b9ba39a05980: Status 404 returned error can't find the container with id 5679037b34cb709b56cc868eac39f2949fab0e99805b38b15e40b9ba39a05980
Jul 31 15:39:03.792896 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.792846    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh" event={"ID":"37165081-ee6e-488e-a49b-95fd82749cea","Type":"ContainerStarted","Data":"7f03dccc9ead21176a14799333eba23e4762094d45b674f43f7539dcda1d44aa"}
Jul 31 15:39:03.793133 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:03.792922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh" event={"ID":"37165081-ee6e-488e-a49b-95fd82749cea","Type":"ContainerStarted","Data":"5679037b34cb709b56cc868eac39f2949fab0e99805b38b15e40b9ba39a05980"}
Jul 31 15:39:07.808912 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:07.808875    2578 generic.go:356] "Generic (PLEG): container finished" podID="37165081-ee6e-488e-a49b-95fd82749cea" containerID="7f03dccc9ead21176a14799333eba23e4762094d45b674f43f7539dcda1d44aa" exitCode=0
Jul 31 15:39:07.809652 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:07.808938    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh" event={"ID":"37165081-ee6e-488e-a49b-95fd82749cea","Type":"ContainerDied","Data":"7f03dccc9ead21176a14799333eba23e4762094d45b674f43f7539dcda1d44aa"}
Jul 31 15:39:08.814733 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:08.814690    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh" event={"ID":"37165081-ee6e-488e-a49b-95fd82749cea","Type":"ContainerStarted","Data":"e68f1e83b52bbdafeaa67422f7675e327ea1547db5d66570d825f88a9ffe4128"}
Jul 31 15:39:08.853226 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:08.853155    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh" podStartSLOduration=2.233448266 podStartE2EDuration="5.853133308s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.619685042s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:39:08.849658391 +0000 UTC m=+3046.858870263" watchObservedRunningTime="2026-07-31 15:39:08.853133308 +0000 UTC m=+3046.862345177"
Jul 31 15:39:09.397909 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:09.397859    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:13.397285 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:13.397233    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:39:13.398875 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:13.398838    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh" podUID="37165081-ee6e-488e-a49b-95fd82749cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 31 15:39:23.397740 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:23.397689    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh" podUID="37165081-ee6e-488e-a49b-95fd82749cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 31 15:39:33.397937 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:33.397888    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh" podUID="37165081-ee6e-488e-a49b-95fd82749cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 31 15:39:43.397935 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:43.397880    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh" podUID="37165081-ee6e-488e-a49b-95fd82749cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 31 15:39:53.398380 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:39:53.398321    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh" podUID="37165081-ee6e-488e-a49b-95fd82749cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 31 15:40:03.398497 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:03.398438    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh" podUID="37165081-ee6e-488e-a49b-95fd82749cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 31 15:40:05.378741 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:05.378692    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"]
Jul 31 15:40:05.379154 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:05.378965    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd" podUID="975ce940-485d-46cf-8760-b35155f58567" containerName="istio-proxy" containerID="cri-o://4c5605e0ef55713ba6ea3afe7be8a68d2b01a3b9b5207d05ab5c1870d4593fbf" gracePeriod=30
Jul 31 15:40:06.964796 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:06.964748    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd" podUID="975ce940-485d-46cf-8760-b35155f58567" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:40:10.621422 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.621386    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:40:10.777968 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.777925    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-istio-envoy\") pod \"975ce940-485d-46cf-8760-b35155f58567\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") "
Jul 31 15:40:10.778167 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.777988    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975ce940-485d-46cf-8760-b35155f58567-istio-podinfo\") pod \"975ce940-485d-46cf-8760-b35155f58567\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") "
Jul 31 15:40:10.778167 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.778020    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975ce940-485d-46cf-8760-b35155f58567-istio-token\") pod \"975ce940-485d-46cf-8760-b35155f58567\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") "
Jul 31 15:40:10.778167 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.778046    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975ce940-485d-46cf-8760-b35155f58567-istiod-ca-cert\") pod \"975ce940-485d-46cf-8760-b35155f58567\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") "
Jul 31 15:40:10.778167 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.778083    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s4vn\" (UniqueName: \"kubernetes.io/projected/975ce940-485d-46cf-8760-b35155f58567-kube-api-access-8s4vn\") pod \"975ce940-485d-46cf-8760-b35155f58567\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") "
Jul 31 15:40:10.778167 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.778128    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-workload-socket\") pod \"975ce940-485d-46cf-8760-b35155f58567\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") "
Jul 31 15:40:10.778167 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.778153    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-workload-certs\") pod \"975ce940-485d-46cf-8760-b35155f58567\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") "
Jul 31 15:40:10.778550 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.778200    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-credential-socket\") pod \"975ce940-485d-46cf-8760-b35155f58567\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") "
Jul 31 15:40:10.778550 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.778244    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-istio-data\") pod \"975ce940-485d-46cf-8760-b35155f58567\" (UID: \"975ce940-485d-46cf-8760-b35155f58567\") "
Jul 31 15:40:10.779586 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.779021    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-istio-data" (OuterVolumeSpecName: "istio-data") pod "975ce940-485d-46cf-8760-b35155f58567" (UID: "975ce940-485d-46cf-8760-b35155f58567"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:40:10.779586 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.779213    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975ce940-485d-46cf-8760-b35155f58567-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "975ce940-485d-46cf-8760-b35155f58567" (UID: "975ce940-485d-46cf-8760-b35155f58567"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:40:10.779586 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.779299    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "975ce940-485d-46cf-8760-b35155f58567" (UID: "975ce940-485d-46cf-8760-b35155f58567"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:40:10.779586 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.779495    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "975ce940-485d-46cf-8760-b35155f58567" (UID: "975ce940-485d-46cf-8760-b35155f58567"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:40:10.779586 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.779553    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "975ce940-485d-46cf-8760-b35155f58567" (UID: "975ce940-485d-46cf-8760-b35155f58567"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:40:10.780810 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.780779    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "975ce940-485d-46cf-8760-b35155f58567" (UID: "975ce940-485d-46cf-8760-b35155f58567"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:40:10.781153 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.781120    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/975ce940-485d-46cf-8760-b35155f58567-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "975ce940-485d-46cf-8760-b35155f58567" (UID: "975ce940-485d-46cf-8760-b35155f58567"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 15:40:10.782320 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.782280    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975ce940-485d-46cf-8760-b35155f58567-kube-api-access-8s4vn" (OuterVolumeSpecName: "kube-api-access-8s4vn") pod "975ce940-485d-46cf-8760-b35155f58567" (UID: "975ce940-485d-46cf-8760-b35155f58567"). InnerVolumeSpecName "kube-api-access-8s4vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:40:10.782785 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.782760    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975ce940-485d-46cf-8760-b35155f58567-istio-token" (OuterVolumeSpecName: "istio-token") pod "975ce940-485d-46cf-8760-b35155f58567" (UID: "975ce940-485d-46cf-8760-b35155f58567"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:40:10.879361 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.879311    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s4vn\" (UniqueName: \"kubernetes.io/projected/975ce940-485d-46cf-8760-b35155f58567-kube-api-access-8s4vn\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:40:10.879361 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.879359    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:40:10.879566 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.879374    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:40:10.879566 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.879383    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:40:10.879566 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.879392    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:40:10.879566 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.879400    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/975ce940-485d-46cf-8760-b35155f58567-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:40:10.879566 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.879408    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/975ce940-485d-46cf-8760-b35155f58567-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:40:10.879566 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.879415    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/975ce940-485d-46cf-8760-b35155f58567-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:40:10.879566 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:10.879422    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/975ce940-485d-46cf-8760-b35155f58567-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:40:11.087881 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:11.087840    2578 generic.go:356] "Generic (PLEG): container finished" podID="975ce940-485d-46cf-8760-b35155f58567" containerID="4c5605e0ef55713ba6ea3afe7be8a68d2b01a3b9b5207d05ab5c1870d4593fbf" exitCode=0
Jul 31 15:40:11.087881 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:11.087883    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"
Jul 31 15:40:11.088180 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:11.087928    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd" event={"ID":"975ce940-485d-46cf-8760-b35155f58567","Type":"ContainerDied","Data":"4c5605e0ef55713ba6ea3afe7be8a68d2b01a3b9b5207d05ab5c1870d4593fbf"}
Jul 31 15:40:11.088180 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:11.087984    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd" event={"ID":"975ce940-485d-46cf-8760-b35155f58567","Type":"ContainerDied","Data":"b91f6b8c011233040d255cb36a6246f3b731bf94476f80cc8c8b1298021492ae"}
Jul 31 15:40:11.088180 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:11.088006    2578 scope.go:117] "RemoveContainer" containerID="4c5605e0ef55713ba6ea3afe7be8a68d2b01a3b9b5207d05ab5c1870d4593fbf"
Jul 31 15:40:11.097598 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:11.097575    2578 scope.go:117] "RemoveContainer" containerID="f935c6a2b542376afe82ca819666a6865986e5cbcbfe54ec656067e8f28d7bfd"
Jul 31 15:40:11.109437 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:11.109411    2578 scope.go:117] "RemoveContainer" containerID="4c5605e0ef55713ba6ea3afe7be8a68d2b01a3b9b5207d05ab5c1870d4593fbf"
Jul 31 15:40:11.109813 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:40:11.109780    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c5605e0ef55713ba6ea3afe7be8a68d2b01a3b9b5207d05ab5c1870d4593fbf\": container with ID starting with 4c5605e0ef55713ba6ea3afe7be8a68d2b01a3b9b5207d05ab5c1870d4593fbf not found: ID does not exist" containerID="4c5605e0ef55713ba6ea3afe7be8a68d2b01a3b9b5207d05ab5c1870d4593fbf"
Jul 31 15:40:11.109930 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:11.109823    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c5605e0ef55713ba6ea3afe7be8a68d2b01a3b9b5207d05ab5c1870d4593fbf"} err="failed to get container status \"4c5605e0ef55713ba6ea3afe7be8a68d2b01a3b9b5207d05ab5c1870d4593fbf\": rpc error: code = NotFound desc = could not find container \"4c5605e0ef55713ba6ea3afe7be8a68d2b01a3b9b5207d05ab5c1870d4593fbf\": container with ID starting with 4c5605e0ef55713ba6ea3afe7be8a68d2b01a3b9b5207d05ab5c1870d4593fbf not found: ID does not exist"
Jul 31 15:40:11.109930 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:11.109849    2578 scope.go:117] "RemoveContainer" containerID="f935c6a2b542376afe82ca819666a6865986e5cbcbfe54ec656067e8f28d7bfd"
Jul 31 15:40:11.110171 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:40:11.110144    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f935c6a2b542376afe82ca819666a6865986e5cbcbfe54ec656067e8f28d7bfd\": container with ID starting with f935c6a2b542376afe82ca819666a6865986e5cbcbfe54ec656067e8f28d7bfd not found: ID does not exist" containerID="f935c6a2b542376afe82ca819666a6865986e5cbcbfe54ec656067e8f28d7bfd"
Jul 31 15:40:11.110265 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:11.110179    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f935c6a2b542376afe82ca819666a6865986e5cbcbfe54ec656067e8f28d7bfd"} err="failed to get container status \"f935c6a2b542376afe82ca819666a6865986e5cbcbfe54ec656067e8f28d7bfd\": rpc error: code = NotFound desc = could not find container \"f935c6a2b542376afe82ca819666a6865986e5cbcbfe54ec656067e8f28d7bfd\": container with ID starting with f935c6a2b542376afe82ca819666a6865986e5cbcbfe54ec656067e8f28d7bfd not found: ID does not exist"
Jul 31 15:40:11.114164 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:11.114138    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"]
Jul 31 15:40:11.120245 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:11.120215    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-8675cdb9d8-f7lgd"]
Jul 31 15:40:12.490689 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:12.490649    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="975ce940-485d-46cf-8760-b35155f58567" path="/var/lib/kubelet/pods/975ce940-485d-46cf-8760-b35155f58567/volumes"
Jul 31 15:40:13.397716 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:13.397672    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh" podUID="37165081-ee6e-488e-a49b-95fd82749cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 31 15:40:17.794089 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.794048    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"]
Jul 31 15:40:17.794739 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.794385    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="975ce940-485d-46cf-8760-b35155f58567" containerName="istio-proxy"
Jul 31 15:40:17.794739 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.794400    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="975ce940-485d-46cf-8760-b35155f58567" containerName="istio-proxy"
Jul 31 15:40:17.794739 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.794456    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="975ce940-485d-46cf-8760-b35155f58567" containerName="istio-proxy"
Jul 31 15:40:17.794739 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.794517    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="975ce940-485d-46cf-8760-b35155f58567" containerName="istio-proxy"
Jul 31 15:40:17.794739 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.794529    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="975ce940-485d-46cf-8760-b35155f58567" containerName="istio-proxy"
Jul 31 15:40:17.794739 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.794574    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="975ce940-485d-46cf-8760-b35155f58567" containerName="istio-proxy"
Jul 31 15:40:17.798956 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.798934    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:17.801786 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.801760    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Jul 31 15:40:17.801957 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.801936    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Jul 31 15:40:17.802070 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.801894    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-6d5jn\""
Jul 31 15:40:17.802070 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.801768    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Jul 31 15:40:17.802070 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.802045    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 31 15:40:17.802070 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.801943    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:40:17.812792 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.812765    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"]
Jul 31 15:40:17.943559 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.943525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:17.943743 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.943572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:17.943743 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.943640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4wpn\" (UniqueName: \"kubernetes.io/projected/685b1b24-941b-46a3-9806-f342830eff97-kube-api-access-s4wpn\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:17.943743 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.943665    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:17.943743 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.943685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:17.943743 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.943703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/685b1b24-941b-46a3-9806-f342830eff97-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:17.943743 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.943730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/685b1b24-941b-46a3-9806-f342830eff97-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:17.943968 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:17.943756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.044879 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.044791    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4wpn\" (UniqueName: \"kubernetes.io/projected/685b1b24-941b-46a3-9806-f342830eff97-kube-api-access-s4wpn\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.044879 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.044831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.044879 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.044857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.045148 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.044886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/685b1b24-941b-46a3-9806-f342830eff97-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.045148 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.044912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/685b1b24-941b-46a3-9806-f342830eff97-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.045148 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.044946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.045148 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.044977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.045148 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.045015    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.045437 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.045302    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.045437 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.045324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.045437 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.045408    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.045723 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.045691    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.045848 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.045811    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/685b1b24-941b-46a3-9806-f342830eff97-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.047552 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.047525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.047703 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.047686    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/685b1b24-941b-46a3-9806-f342830eff97-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.053289 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.053269    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4wpn\" (UniqueName: \"kubernetes.io/projected/685b1b24-941b-46a3-9806-f342830eff97-kube-api-access-s4wpn\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.112364 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.112294    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:18.259580 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:18.259544    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"]
Jul 31 15:40:18.261500 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:40:18.261469    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685b1b24_941b_46a3_9806_f342830eff97.slice/crio-0319760df7f81381500d466431f1eaceb3952d27d9396f459df024cc44474236 WatchSource:0}: Error finding container 0319760df7f81381500d466431f1eaceb3952d27d9396f459df024cc44474236: Status 404 returned error can't find the container with id 0319760df7f81381500d466431f1eaceb3952d27d9396f459df024cc44474236
Jul 31 15:40:19.123465 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:19.123422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" event={"ID":"685b1b24-941b-46a3-9806-f342830eff97","Type":"ContainerStarted","Data":"933afb65f87f400b28cdcee0e32062cb2ef49d1c21e1a3f5935efa62c9b1b895"}
Jul 31 15:40:19.123931 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:19.123480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" event={"ID":"685b1b24-941b-46a3-9806-f342830eff97","Type":"ContainerStarted","Data":"0319760df7f81381500d466431f1eaceb3952d27d9396f459df024cc44474236"}
Jul 31 15:40:21.133278 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:21.133242    2578 generic.go:356] "Generic (PLEG): container finished" podID="685b1b24-941b-46a3-9806-f342830eff97" containerID="933afb65f87f400b28cdcee0e32062cb2ef49d1c21e1a3f5935efa62c9b1b895" exitCode=0
Jul 31 15:40:21.133701 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:21.133295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" event={"ID":"685b1b24-941b-46a3-9806-f342830eff97","Type":"ContainerDied","Data":"933afb65f87f400b28cdcee0e32062cb2ef49d1c21e1a3f5935efa62c9b1b895"}
Jul 31 15:40:22.142031 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:22.141985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" event={"ID":"685b1b24-941b-46a3-9806-f342830eff97","Type":"ContainerStarted","Data":"7f413150ef5ff1aed4023e27bfeeaeb442493fe198c3f6d523c6de2280d68888"}
Jul 31 15:40:22.171367 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:22.171282    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" podStartSLOduration=2.832257511 podStartE2EDuration="5.171256611s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.3389991s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:40:22.169820578 +0000 UTC m=+3120.179032455" watchObservedRunningTime="2026-07-31 15:40:22.171256611 +0000 UTC m=+3120.180468493"
Jul 31 15:40:23.398133 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:23.398081    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh" podUID="37165081-ee6e-488e-a49b-95fd82749cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 31 15:40:28.112873 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:28.112819    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:28.113436 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:28.112895    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:40:28.114295 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:28.114260    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 31 15:40:33.413424 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:33.413384    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:40:33.425462 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:33.425436    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:40:38.112939 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:38.112888    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 31 15:40:48.113578 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:48.113482    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 31 15:40:58.113215 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:40:58.113170    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 31 15:41:08.113409 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:08.113336    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 31 15:41:18.112962 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:18.112911    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 31 15:41:28.112989 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:28.112936    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 31 15:41:33.787963 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:33.787932    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh_37165081-ee6e-488e-a49b-95fd82749cea/storage-initializer/0.log"
Jul 31 15:41:33.799322 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:33.799292    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh_37165081-ee6e-488e-a49b-95fd82749cea/main/0.log"
Jul 31 15:41:34.952736 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:34.952639    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"]
Jul 31 15:41:38.113735 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:38.113688    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 31 15:41:48.113738 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:48.113689    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 31 15:41:49.974791 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:49.974740    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh" podUID="37165081-ee6e-488e-a49b-95fd82749cea" containerName="main" containerID="cri-o://e68f1e83b52bbdafeaa67422f7675e327ea1547db5d66570d825f88a9ffe4128" gracePeriod=45
Jul 31 15:41:50.414664 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:50.414560    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh" podUID="37165081-ee6e-488e-a49b-95fd82749cea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 31 15:41:50.930899 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:50.930867    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:41:51.104980 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.104940    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-kserve-provision-location\") pod \"37165081-ee6e-488e-a49b-95fd82749cea\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") "
Jul 31 15:41:51.105499 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.104995    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-home\") pod \"37165081-ee6e-488e-a49b-95fd82749cea\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") "
Jul 31 15:41:51.105499 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.105050    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-model-cache\") pod \"37165081-ee6e-488e-a49b-95fd82749cea\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") "
Jul 31 15:41:51.105499 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.105081    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2msg\" (UniqueName: \"kubernetes.io/projected/37165081-ee6e-488e-a49b-95fd82749cea-kube-api-access-x2msg\") pod \"37165081-ee6e-488e-a49b-95fd82749cea\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") "
Jul 31 15:41:51.105499 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.105115    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37165081-ee6e-488e-a49b-95fd82749cea-tls-certs\") pod \"37165081-ee6e-488e-a49b-95fd82749cea\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") "
Jul 31 15:41:51.105499 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.105160    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-dshm\") pod \"37165081-ee6e-488e-a49b-95fd82749cea\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") "
Jul 31 15:41:51.105499 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.105269    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-model-cache" (OuterVolumeSpecName: "model-cache") pod "37165081-ee6e-488e-a49b-95fd82749cea" (UID: "37165081-ee6e-488e-a49b-95fd82749cea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:41:51.105499 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.105297    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37165081-ee6e-488e-a49b-95fd82749cea-cabundle-cert\") pod \"37165081-ee6e-488e-a49b-95fd82749cea\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") "
Jul 31 15:41:51.105499 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.105370    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-tmp-dir\") pod \"37165081-ee6e-488e-a49b-95fd82749cea\" (UID: \"37165081-ee6e-488e-a49b-95fd82749cea\") "
Jul 31 15:41:51.105499 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.105452    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-home" (OuterVolumeSpecName: "home") pod "37165081-ee6e-488e-a49b-95fd82749cea" (UID: "37165081-ee6e-488e-a49b-95fd82749cea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:41:51.105973 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.105642    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37165081-ee6e-488e-a49b-95fd82749cea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "37165081-ee6e-488e-a49b-95fd82749cea" (UID: "37165081-ee6e-488e-a49b-95fd82749cea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:41:51.105973 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.105791    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:41:51.105973 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.105810    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:41:51.105973 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.105824    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37165081-ee6e-488e-a49b-95fd82749cea-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:41:51.107640 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.107602    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-dshm" (OuterVolumeSpecName: "dshm") pod "37165081-ee6e-488e-a49b-95fd82749cea" (UID: "37165081-ee6e-488e-a49b-95fd82749cea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:41:51.107742 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.107706    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37165081-ee6e-488e-a49b-95fd82749cea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37165081-ee6e-488e-a49b-95fd82749cea" (UID: "37165081-ee6e-488e-a49b-95fd82749cea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:41:51.107969 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.107945    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37165081-ee6e-488e-a49b-95fd82749cea-kube-api-access-x2msg" (OuterVolumeSpecName: "kube-api-access-x2msg") pod "37165081-ee6e-488e-a49b-95fd82749cea" (UID: "37165081-ee6e-488e-a49b-95fd82749cea"). InnerVolumeSpecName "kube-api-access-x2msg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:41:51.122832 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.122798    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37165081-ee6e-488e-a49b-95fd82749cea" (UID: "37165081-ee6e-488e-a49b-95fd82749cea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:41:51.131618 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.131589    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37165081-ee6e-488e-a49b-95fd82749cea" (UID: "37165081-ee6e-488e-a49b-95fd82749cea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:41:51.206569 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.206530    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:41:51.206569 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.206568    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2msg\" (UniqueName: \"kubernetes.io/projected/37165081-ee6e-488e-a49b-95fd82749cea-kube-api-access-x2msg\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:41:51.206747 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.206582    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37165081-ee6e-488e-a49b-95fd82749cea-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:41:51.206747 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.206596    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:41:51.206747 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.206607    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37165081-ee6e-488e-a49b-95fd82749cea-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:41:51.590659 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.590621    2578 generic.go:356] "Generic (PLEG): container finished" podID="37165081-ee6e-488e-a49b-95fd82749cea" containerID="e68f1e83b52bbdafeaa67422f7675e327ea1547db5d66570d825f88a9ffe4128" exitCode=0
Jul 31 15:41:51.590873 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.590682    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"
Jul 31 15:41:51.590873 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.590711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh" event={"ID":"37165081-ee6e-488e-a49b-95fd82749cea","Type":"ContainerDied","Data":"e68f1e83b52bbdafeaa67422f7675e327ea1547db5d66570d825f88a9ffe4128"}
Jul 31 15:41:51.590873 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.590766    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh" event={"ID":"37165081-ee6e-488e-a49b-95fd82749cea","Type":"ContainerDied","Data":"5679037b34cb709b56cc868eac39f2949fab0e99805b38b15e40b9ba39a05980"}
Jul 31 15:41:51.590873 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.590789    2578 scope.go:117] "RemoveContainer" containerID="e68f1e83b52bbdafeaa67422f7675e327ea1547db5d66570d825f88a9ffe4128"
Jul 31 15:41:51.600686 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.600327    2578 scope.go:117] "RemoveContainer" containerID="7f03dccc9ead21176a14799333eba23e4762094d45b674f43f7539dcda1d44aa"
Jul 31 15:41:51.608519 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.608495    2578 scope.go:117] "RemoveContainer" containerID="e68f1e83b52bbdafeaa67422f7675e327ea1547db5d66570d825f88a9ffe4128"
Jul 31 15:41:51.608772 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:41:51.608754    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68f1e83b52bbdafeaa67422f7675e327ea1547db5d66570d825f88a9ffe4128\": container with ID starting with e68f1e83b52bbdafeaa67422f7675e327ea1547db5d66570d825f88a9ffe4128 not found: ID does not exist" containerID="e68f1e83b52bbdafeaa67422f7675e327ea1547db5d66570d825f88a9ffe4128"
Jul 31 15:41:51.608845 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.608780    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68f1e83b52bbdafeaa67422f7675e327ea1547db5d66570d825f88a9ffe4128"} err="failed to get container status \"e68f1e83b52bbdafeaa67422f7675e327ea1547db5d66570d825f88a9ffe4128\": rpc error: code = NotFound desc = could not find container \"e68f1e83b52bbdafeaa67422f7675e327ea1547db5d66570d825f88a9ffe4128\": container with ID starting with e68f1e83b52bbdafeaa67422f7675e327ea1547db5d66570d825f88a9ffe4128 not found: ID does not exist"
Jul 31 15:41:51.608845 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.608797    2578 scope.go:117] "RemoveContainer" containerID="7f03dccc9ead21176a14799333eba23e4762094d45b674f43f7539dcda1d44aa"
Jul 31 15:41:51.609024 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:41:51.609006    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f03dccc9ead21176a14799333eba23e4762094d45b674f43f7539dcda1d44aa\": container with ID starting with 7f03dccc9ead21176a14799333eba23e4762094d45b674f43f7539dcda1d44aa not found: ID does not exist" containerID="7f03dccc9ead21176a14799333eba23e4762094d45b674f43f7539dcda1d44aa"
Jul 31 15:41:51.609080 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.609027    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f03dccc9ead21176a14799333eba23e4762094d45b674f43f7539dcda1d44aa"} err="failed to get container status \"7f03dccc9ead21176a14799333eba23e4762094d45b674f43f7539dcda1d44aa\": rpc error: code = NotFound desc = could not find container \"7f03dccc9ead21176a14799333eba23e4762094d45b674f43f7539dcda1d44aa\": container with ID starting with 7f03dccc9ead21176a14799333eba23e4762094d45b674f43f7539dcda1d44aa not found: ID does not exist"
Jul 31 15:41:51.615481 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.615458    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"]
Jul 31 15:41:51.620485 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:51.620464    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cc5d5d599-r4jvh"]
Jul 31 15:41:52.489615 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:52.489572    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37165081-ee6e-488e-a49b-95fd82749cea" path="/var/lib/kubelet/pods/37165081-ee6e-488e-a49b-95fd82749cea/volumes"
Jul 31 15:41:58.113120 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.113074    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 31 15:41:58.521045 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.521012    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"]
Jul 31 15:41:58.521289 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.521277    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37165081-ee6e-488e-a49b-95fd82749cea" containerName="main"
Jul 31 15:41:58.521335 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.521290    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="37165081-ee6e-488e-a49b-95fd82749cea" containerName="main"
Jul 31 15:41:58.521335 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.521299    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37165081-ee6e-488e-a49b-95fd82749cea" containerName="storage-initializer"
Jul 31 15:41:58.521335 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.521305    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="37165081-ee6e-488e-a49b-95fd82749cea" containerName="storage-initializer"
Jul 31 15:41:58.521465 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.521388    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="37165081-ee6e-488e-a49b-95fd82749cea" containerName="main"
Jul 31 15:41:58.524259 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.524240    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.526931 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.526898    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Jul 31 15:41:58.527057 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.526961    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Jul 31 15:41:58.527057 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.526960    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:41:58.527323 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.527307    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-qx2nq\""
Jul 31 15:41:58.527402 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.527323    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 31 15:41:58.527402 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.527340    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Jul 31 15:41:58.533362 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.533329    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"]
Jul 31 15:41:58.561178 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.561145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-dshm\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.561330 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.561195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-kserve-provision-location\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.561330 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.561230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-model-cache\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.561330 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.561277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/483378f4-dcc6-419b-932e-fc655a7e5ec0-tls-certs\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.561513 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.561370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-tmp-dir\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.561513 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.561408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhfxv\" (UniqueName: \"kubernetes.io/projected/483378f4-dcc6-419b-932e-fc655a7e5ec0-kube-api-access-lhfxv\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.561513 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.561433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/483378f4-dcc6-419b-932e-fc655a7e5ec0-cabundle-cert\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.561513 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.561497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-home\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.662186 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.662143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhfxv\" (UniqueName: \"kubernetes.io/projected/483378f4-dcc6-419b-932e-fc655a7e5ec0-kube-api-access-lhfxv\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.662405 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.662192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/483378f4-dcc6-419b-932e-fc655a7e5ec0-cabundle-cert\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.662405 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.662229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-home\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.662405 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.662271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-dshm\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.662405 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.662299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-kserve-provision-location\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.662637 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.662415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-model-cache\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.662637 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.662481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/483378f4-dcc6-419b-932e-fc655a7e5ec0-tls-certs\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.662637 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.662514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-tmp-dir\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.662792 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.662762    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-home\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.662943 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.662921    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-tmp-dir\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.663000 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.662941    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-model-cache\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.663087 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.663063    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-kserve-provision-location\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.663164 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.663114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/483378f4-dcc6-419b-932e-fc655a7e5ec0-cabundle-cert\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.664818 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.664796    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-dshm\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.665060 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.665042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/483378f4-dcc6-419b-932e-fc655a7e5ec0-tls-certs\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.671142 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.671117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhfxv\" (UniqueName: \"kubernetes.io/projected/483378f4-dcc6-419b-932e-fc655a7e5ec0-kube-api-access-lhfxv\") pod \"tls-verification-test-kserve-7fb9d5c559-scwzx\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.836075 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.835960    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:41:58.987684 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:41:58.987650    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483378f4_dcc6_419b_932e_fc655a7e5ec0.slice/crio-ba32ccbb050810538654fece7d42990e2dfa6f99c10051a2ad5a8b2fc2a6f166 WatchSource:0}: Error finding container ba32ccbb050810538654fece7d42990e2dfa6f99c10051a2ad5a8b2fc2a6f166: Status 404 returned error can't find the container with id ba32ccbb050810538654fece7d42990e2dfa6f99c10051a2ad5a8b2fc2a6f166
Jul 31 15:41:58.989053 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:58.989023    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"]
Jul 31 15:41:59.634435 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:59.634392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" event={"ID":"483378f4-dcc6-419b-932e-fc655a7e5ec0","Type":"ContainerStarted","Data":"26892e3d48e69185101986f555e67c468141a34e500ed6b178317918e614c40a"}
Jul 31 15:41:59.635023 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:41:59.634464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" event={"ID":"483378f4-dcc6-419b-932e-fc655a7e5ec0","Type":"ContainerStarted","Data":"ba32ccbb050810538654fece7d42990e2dfa6f99c10051a2ad5a8b2fc2a6f166"}
Jul 31 15:42:01.647473 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:01.647427    2578 generic.go:356] "Generic (PLEG): container finished" podID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerID="26892e3d48e69185101986f555e67c468141a34e500ed6b178317918e614c40a" exitCode=0
Jul 31 15:42:01.647846 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:01.647499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" event={"ID":"483378f4-dcc6-419b-932e-fc655a7e5ec0","Type":"ContainerDied","Data":"26892e3d48e69185101986f555e67c468141a34e500ed6b178317918e614c40a"}
Jul 31 15:42:02.653536 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:02.653499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" event={"ID":"483378f4-dcc6-419b-932e-fc655a7e5ec0","Type":"ContainerStarted","Data":"7d5034e1aaa87a995ce66d41af30f9535595d7989202b60b43a5437568d9bc96"}
Jul 31 15:42:02.678096 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:02.678036    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" podStartSLOduration=2.245888754 podStartE2EDuration="4.67801757s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.432128816s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:42:02.676292693 +0000 UTC m=+3220.685504563" watchObservedRunningTime="2026-07-31 15:42:02.67801757 +0000 UTC m=+3220.687229441"
Jul 31 15:42:02.837204 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:02.837151    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:42:08.113924 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:08.113866    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 31 15:42:08.837075 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:08.837034    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:42:08.838675 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:08.838642    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 31 15:42:18.113442 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:18.113386    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 31 15:42:18.836978 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:18.836931    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 31 15:42:28.123672 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:28.123632    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:42:28.131632 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:28.131601    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:42:28.837270 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:28.837213    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 31 15:42:36.897054 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:36.897007    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"]
Jul 31 15:42:38.836636 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:38.836584    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 31 15:42:48.837144 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:48.837091    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 31 15:42:51.919777 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:51.919713    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="main" containerID="cri-o://7f413150ef5ff1aed4023e27bfeeaeb442493fe198c3f6d523c6de2280d68888" gracePeriod=45
Jul 31 15:42:52.662342 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.662315    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:42:52.745211 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.745177    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-home\") pod \"685b1b24-941b-46a3-9806-f342830eff97\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") "
Jul 31 15:42:52.745414 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.745218    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-model-cache\") pod \"685b1b24-941b-46a3-9806-f342830eff97\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") "
Jul 31 15:42:52.745414 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.745240    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/685b1b24-941b-46a3-9806-f342830eff97-tls-certs\") pod \"685b1b24-941b-46a3-9806-f342830eff97\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") "
Jul 31 15:42:52.745414 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.745272    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-dshm\") pod \"685b1b24-941b-46a3-9806-f342830eff97\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") "
Jul 31 15:42:52.745414 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.745322    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4wpn\" (UniqueName: \"kubernetes.io/projected/685b1b24-941b-46a3-9806-f342830eff97-kube-api-access-s4wpn\") pod \"685b1b24-941b-46a3-9806-f342830eff97\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") "
Jul 31 15:42:52.745414 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.745361    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/685b1b24-941b-46a3-9806-f342830eff97-cabundle-cert\") pod \"685b1b24-941b-46a3-9806-f342830eff97\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") "
Jul 31 15:42:52.745414 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.745402    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-kserve-provision-location\") pod \"685b1b24-941b-46a3-9806-f342830eff97\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") "
Jul 31 15:42:52.745753 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.745459    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-tmp-dir\") pod \"685b1b24-941b-46a3-9806-f342830eff97\" (UID: \"685b1b24-941b-46a3-9806-f342830eff97\") "
Jul 31 15:42:52.745753 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.745487    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-model-cache" (OuterVolumeSpecName: "model-cache") pod "685b1b24-941b-46a3-9806-f342830eff97" (UID: "685b1b24-941b-46a3-9806-f342830eff97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:42:52.745753 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.745695    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:42:52.745753 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.745750    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/685b1b24-941b-46a3-9806-f342830eff97-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "685b1b24-941b-46a3-9806-f342830eff97" (UID: "685b1b24-941b-46a3-9806-f342830eff97"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:42:52.746075 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.746041    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-home" (OuterVolumeSpecName: "home") pod "685b1b24-941b-46a3-9806-f342830eff97" (UID: "685b1b24-941b-46a3-9806-f342830eff97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:42:52.747684 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.747655    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-dshm" (OuterVolumeSpecName: "dshm") pod "685b1b24-941b-46a3-9806-f342830eff97" (UID: "685b1b24-941b-46a3-9806-f342830eff97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:42:52.747948 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.747921    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/685b1b24-941b-46a3-9806-f342830eff97-kube-api-access-s4wpn" (OuterVolumeSpecName: "kube-api-access-s4wpn") pod "685b1b24-941b-46a3-9806-f342830eff97" (UID: "685b1b24-941b-46a3-9806-f342830eff97"). InnerVolumeSpecName "kube-api-access-s4wpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:42:52.748140 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.748119    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/685b1b24-941b-46a3-9806-f342830eff97-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "685b1b24-941b-46a3-9806-f342830eff97" (UID: "685b1b24-941b-46a3-9806-f342830eff97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:42:52.764322 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.764296    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "685b1b24-941b-46a3-9806-f342830eff97" (UID: "685b1b24-941b-46a3-9806-f342830eff97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:42:52.772019 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.771997    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "685b1b24-941b-46a3-9806-f342830eff97" (UID: "685b1b24-941b-46a3-9806-f342830eff97"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:42:52.846339 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.846284    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:42:52.846339 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.846338    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:42:52.846579 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.846372    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/685b1b24-941b-46a3-9806-f342830eff97-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:42:52.846579 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.846387    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:42:52.846579 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.846399    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4wpn\" (UniqueName: \"kubernetes.io/projected/685b1b24-941b-46a3-9806-f342830eff97-kube-api-access-s4wpn\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:42:52.846579 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.846410    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/685b1b24-941b-46a3-9806-f342830eff97-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:42:52.846579 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.846422    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/685b1b24-941b-46a3-9806-f342830eff97-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:42:52.869273 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.869235    2578 generic.go:356] "Generic (PLEG): container finished" podID="685b1b24-941b-46a3-9806-f342830eff97" containerID="7f413150ef5ff1aed4023e27bfeeaeb442493fe198c3f6d523c6de2280d68888" exitCode=0
Jul 31 15:42:52.869436 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.869301    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"
Jul 31 15:42:52.869436 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.869320    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" event={"ID":"685b1b24-941b-46a3-9806-f342830eff97","Type":"ContainerDied","Data":"7f413150ef5ff1aed4023e27bfeeaeb442493fe198c3f6d523c6de2280d68888"}
Jul 31 15:42:52.869436 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.869399    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx" event={"ID":"685b1b24-941b-46a3-9806-f342830eff97","Type":"ContainerDied","Data":"0319760df7f81381500d466431f1eaceb3952d27d9396f459df024cc44474236"}
Jul 31 15:42:52.869436 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.869432    2578 scope.go:117] "RemoveContainer" containerID="7f413150ef5ff1aed4023e27bfeeaeb442493fe198c3f6d523c6de2280d68888"
Jul 31 15:42:52.878778 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.878761    2578 scope.go:117] "RemoveContainer" containerID="933afb65f87f400b28cdcee0e32062cb2ef49d1c21e1a3f5935efa62c9b1b895"
Jul 31 15:42:52.887685 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.887649    2578 scope.go:117] "RemoveContainer" containerID="7f413150ef5ff1aed4023e27bfeeaeb442493fe198c3f6d523c6de2280d68888"
Jul 31 15:42:52.888088 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:42:52.888061    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f413150ef5ff1aed4023e27bfeeaeb442493fe198c3f6d523c6de2280d68888\": container with ID starting with 7f413150ef5ff1aed4023e27bfeeaeb442493fe198c3f6d523c6de2280d68888 not found: ID does not exist" containerID="7f413150ef5ff1aed4023e27bfeeaeb442493fe198c3f6d523c6de2280d68888"
Jul 31 15:42:52.888167 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.888098    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f413150ef5ff1aed4023e27bfeeaeb442493fe198c3f6d523c6de2280d68888"} err="failed to get container status \"7f413150ef5ff1aed4023e27bfeeaeb442493fe198c3f6d523c6de2280d68888\": rpc error: code = NotFound desc = could not find container \"7f413150ef5ff1aed4023e27bfeeaeb442493fe198c3f6d523c6de2280d68888\": container with ID starting with 7f413150ef5ff1aed4023e27bfeeaeb442493fe198c3f6d523c6de2280d68888 not found: ID does not exist"
Jul 31 15:42:52.888167 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.888120    2578 scope.go:117] "RemoveContainer" containerID="933afb65f87f400b28cdcee0e32062cb2ef49d1c21e1a3f5935efa62c9b1b895"
Jul 31 15:42:52.888416 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:42:52.888391    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933afb65f87f400b28cdcee0e32062cb2ef49d1c21e1a3f5935efa62c9b1b895\": container with ID starting with 933afb65f87f400b28cdcee0e32062cb2ef49d1c21e1a3f5935efa62c9b1b895 not found: ID does not exist" containerID="933afb65f87f400b28cdcee0e32062cb2ef49d1c21e1a3f5935efa62c9b1b895"
Jul 31 15:42:52.888520 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.888420    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933afb65f87f400b28cdcee0e32062cb2ef49d1c21e1a3f5935efa62c9b1b895"} err="failed to get container status \"933afb65f87f400b28cdcee0e32062cb2ef49d1c21e1a3f5935efa62c9b1b895\": rpc error: code = NotFound desc = could not find container \"933afb65f87f400b28cdcee0e32062cb2ef49d1c21e1a3f5935efa62c9b1b895\": container with ID starting with 933afb65f87f400b28cdcee0e32062cb2ef49d1c21e1a3f5935efa62c9b1b895 not found: ID does not exist"
Jul 31 15:42:52.893208 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.893187    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"]
Jul 31 15:42:52.899136 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:52.899113    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7trwzx"]
Jul 31 15:42:54.489889 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:54.489855    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="685b1b24-941b-46a3-9806-f342830eff97" path="/var/lib/kubelet/pods/685b1b24-941b-46a3-9806-f342830eff97/volumes"
Jul 31 15:42:58.836889 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:42:58.836844    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 31 15:43:08.188872 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.188839    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"]
Jul 31 15:43:08.189328 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.189118    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="main"
Jul 31 15:43:08.189328 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.189130    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="main"
Jul 31 15:43:08.189328 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.189139    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="storage-initializer"
Jul 31 15:43:08.189328 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.189146    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="storage-initializer"
Jul 31 15:43:08.189328 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.189193    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="685b1b24-941b-46a3-9806-f342830eff97" containerName="main"
Jul 31 15:43:08.191774 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.191756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.194714 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.194692    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:43:08.195158 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.195137    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Jul 31 15:43:08.195336 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.195306    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Jul 31 15:43:08.195442 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.195342    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-bmxgl\""
Jul 31 15:43:08.195442 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.195363    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Jul 31 15:43:08.195442 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.195406    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 31 15:43:08.206809 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.206785    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"]
Jul 31 15:43:08.279116 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.279081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.279116 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.279118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75261747-04ae-4586-8d24-0757152500d9-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.279323 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.279147    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.279323 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.279185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75261747-04ae-4586-8d24-0757152500d9-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.279323 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.279221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.279323 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.279248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.279323 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.279265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.279323 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.279281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn54m\" (UniqueName: \"kubernetes.io/projected/75261747-04ae-4586-8d24-0757152500d9-kube-api-access-dn54m\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.379963 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.379919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.379963 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.379966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.380224 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.379986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.380224 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.380002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn54m\" (UniqueName: \"kubernetes.io/projected/75261747-04ae-4586-8d24-0757152500d9-kube-api-access-dn54m\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.380224 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.380044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.380224 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.380070    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75261747-04ae-4586-8d24-0757152500d9-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.380224 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.380088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.380477 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.380443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75261747-04ae-4586-8d24-0757152500d9-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.380549 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.380478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.380549 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.380514    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.380668 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.380637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.380728 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.380680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.381188 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.381160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75261747-04ae-4586-8d24-0757152500d9-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.382501 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.382478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.382780 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.382761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75261747-04ae-4586-8d24-0757152500d9-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.389674 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.389639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn54m\" (UniqueName: \"kubernetes.io/projected/75261747-04ae-4586-8d24-0757152500d9-kube-api-access-dn54m\") pod \"custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.502466 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.502436    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:08.640252 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.640216    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"]
Jul 31 15:43:08.643743 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:43:08.643713    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75261747_04ae_4586_8d24_0757152500d9.slice/crio-15551c29d3cccbfa628fa8f15c81091ad7ab27f39b8923a1b70f51a6c2f86061 WatchSource:0}: Error finding container 15551c29d3cccbfa628fa8f15c81091ad7ab27f39b8923a1b70f51a6c2f86061: Status 404 returned error can't find the container with id 15551c29d3cccbfa628fa8f15c81091ad7ab27f39b8923a1b70f51a6c2f86061
Jul 31 15:43:08.837396 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.837288    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 31 15:43:08.935482 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.935437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" event={"ID":"75261747-04ae-4586-8d24-0757152500d9","Type":"ContainerStarted","Data":"72ed07c48a1e51dba7932bd849a5e18a57e29bb0975de6cfc3a7612f6b58f081"}
Jul 31 15:43:08.935646 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:08.935510    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" event={"ID":"75261747-04ae-4586-8d24-0757152500d9","Type":"ContainerStarted","Data":"15551c29d3cccbfa628fa8f15c81091ad7ab27f39b8923a1b70f51a6c2f86061"}
Jul 31 15:43:11.951007 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:11.950969    2578 generic.go:356] "Generic (PLEG): container finished" podID="75261747-04ae-4586-8d24-0757152500d9" containerID="72ed07c48a1e51dba7932bd849a5e18a57e29bb0975de6cfc3a7612f6b58f081" exitCode=0
Jul 31 15:43:11.951007 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:11.951010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" event={"ID":"75261747-04ae-4586-8d24-0757152500d9","Type":"ContainerDied","Data":"72ed07c48a1e51dba7932bd849a5e18a57e29bb0975de6cfc3a7612f6b58f081"}
Jul 31 15:43:12.957896 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:12.957859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" event={"ID":"75261747-04ae-4586-8d24-0757152500d9","Type":"ContainerStarted","Data":"fa13d677375c4e1d176521893978c3c304882d8c0725da27aa5b1e8564665e25"}
Jul 31 15:43:12.980789 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:12.980736    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" podStartSLOduration=2.686996383 podStartE2EDuration="4.980718529s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.293722146s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:43:12.97884435 +0000 UTC m=+3290.988056240" watchObservedRunningTime="2026-07-31 15:43:12.980718529 +0000 UTC m=+3290.989930397"
Jul 31 15:43:18.503505 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:18.503468    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:18.503894 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:18.503542    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:43:18.505294 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:18.505266    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" podUID="75261747-04ae-4586-8d24-0757152500d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 31 15:43:18.837369 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:18.837244    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 31 15:43:28.503756 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:28.503708    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" podUID="75261747-04ae-4586-8d24-0757152500d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 31 15:43:28.837226 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:28.837125    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 31 15:43:38.503558 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:38.503506    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" podUID="75261747-04ae-4586-8d24-0757152500d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 31 15:43:38.836742 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:38.836636    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 31 15:43:48.503709 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:48.503605    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" podUID="75261747-04ae-4586-8d24-0757152500d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 31 15:43:48.837556 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:48.837452    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Jul 31 15:43:58.503813 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:58.503762    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" podUID="75261747-04ae-4586-8d24-0757152500d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 31 15:43:58.852566 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:58.852477    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:43:58.864996 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:43:58.864967    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:44:03.766715 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:03.766622    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"]
Jul 31 15:44:08.503237 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:08.503194    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" podUID="75261747-04ae-4586-8d24-0757152500d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 31 15:44:18.503447 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:18.503393    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" podUID="75261747-04ae-4586-8d24-0757152500d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 31 15:44:18.792041 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:18.791920    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerName="main" containerID="cri-o://7d5034e1aaa87a995ce66d41af30f9535595d7989202b60b43a5437568d9bc96" gracePeriod=45
Jul 31 15:44:19.531265 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.531237    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:44:19.700127 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.700007    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhfxv\" (UniqueName: \"kubernetes.io/projected/483378f4-dcc6-419b-932e-fc655a7e5ec0-kube-api-access-lhfxv\") pod \"483378f4-dcc6-419b-932e-fc655a7e5ec0\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") "
Jul 31 15:44:19.700127 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.700082    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/483378f4-dcc6-419b-932e-fc655a7e5ec0-cabundle-cert\") pod \"483378f4-dcc6-419b-932e-fc655a7e5ec0\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") "
Jul 31 15:44:19.700127 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.700119    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/483378f4-dcc6-419b-932e-fc655a7e5ec0-tls-certs\") pod \"483378f4-dcc6-419b-932e-fc655a7e5ec0\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") "
Jul 31 15:44:19.700127 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.700137    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-dshm\") pod \"483378f4-dcc6-419b-932e-fc655a7e5ec0\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") "
Jul 31 15:44:19.700509 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.700178    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-tmp-dir\") pod \"483378f4-dcc6-419b-932e-fc655a7e5ec0\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") "
Jul 31 15:44:19.700509 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.700210    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-model-cache\") pod \"483378f4-dcc6-419b-932e-fc655a7e5ec0\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") "
Jul 31 15:44:19.700509 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.700238    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-kserve-provision-location\") pod \"483378f4-dcc6-419b-932e-fc655a7e5ec0\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") "
Jul 31 15:44:19.700509 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.700261    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-home\") pod \"483378f4-dcc6-419b-932e-fc655a7e5ec0\" (UID: \"483378f4-dcc6-419b-932e-fc655a7e5ec0\") "
Jul 31 15:44:19.700718 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.700576    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/483378f4-dcc6-419b-932e-fc655a7e5ec0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "483378f4-dcc6-419b-932e-fc655a7e5ec0" (UID: "483378f4-dcc6-419b-932e-fc655a7e5ec0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:44:19.701183 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.701142    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-model-cache" (OuterVolumeSpecName: "model-cache") pod "483378f4-dcc6-419b-932e-fc655a7e5ec0" (UID: "483378f4-dcc6-419b-932e-fc655a7e5ec0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:44:19.701319 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.701189    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-home" (OuterVolumeSpecName: "home") pod "483378f4-dcc6-419b-932e-fc655a7e5ec0" (UID: "483378f4-dcc6-419b-932e-fc655a7e5ec0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:44:19.702995 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.702965    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/483378f4-dcc6-419b-932e-fc655a7e5ec0-kube-api-access-lhfxv" (OuterVolumeSpecName: "kube-api-access-lhfxv") pod "483378f4-dcc6-419b-932e-fc655a7e5ec0" (UID: "483378f4-dcc6-419b-932e-fc655a7e5ec0"). InnerVolumeSpecName "kube-api-access-lhfxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:44:19.703119 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.703021    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-dshm" (OuterVolumeSpecName: "dshm") pod "483378f4-dcc6-419b-932e-fc655a7e5ec0" (UID: "483378f4-dcc6-419b-932e-fc655a7e5ec0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:44:19.703119 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.703107    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/483378f4-dcc6-419b-932e-fc655a7e5ec0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "483378f4-dcc6-419b-932e-fc655a7e5ec0" (UID: "483378f4-dcc6-419b-932e-fc655a7e5ec0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:44:19.719365 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.719292    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "483378f4-dcc6-419b-932e-fc655a7e5ec0" (UID: "483378f4-dcc6-419b-932e-fc655a7e5ec0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:44:19.729005 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.728970    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "483378f4-dcc6-419b-932e-fc655a7e5ec0" (UID: "483378f4-dcc6-419b-932e-fc655a7e5ec0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:44:19.801396 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.801339    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhfxv\" (UniqueName: \"kubernetes.io/projected/483378f4-dcc6-419b-932e-fc655a7e5ec0-kube-api-access-lhfxv\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:44:19.801396 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.801384    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/483378f4-dcc6-419b-932e-fc655a7e5ec0-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:44:19.801396 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.801395    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/483378f4-dcc6-419b-932e-fc655a7e5ec0-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:44:19.801396 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.801403    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:44:19.801720 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.801412    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:44:19.801720 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.801421    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:44:19.801720 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.801429    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:44:19.801720 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:19.801437    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/483378f4-dcc6-419b-932e-fc655a7e5ec0-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:44:20.250228 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:20.250188    2578 generic.go:356] "Generic (PLEG): container finished" podID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerID="7d5034e1aaa87a995ce66d41af30f9535595d7989202b60b43a5437568d9bc96" exitCode=0
Jul 31 15:44:20.250458 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:20.250234    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"
Jul 31 15:44:20.250458 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:20.250270    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" event={"ID":"483378f4-dcc6-419b-932e-fc655a7e5ec0","Type":"ContainerDied","Data":"7d5034e1aaa87a995ce66d41af30f9535595d7989202b60b43a5437568d9bc96"}
Jul 31 15:44:20.250458 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:20.250327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx" event={"ID":"483378f4-dcc6-419b-932e-fc655a7e5ec0","Type":"ContainerDied","Data":"ba32ccbb050810538654fece7d42990e2dfa6f99c10051a2ad5a8b2fc2a6f166"}
Jul 31 15:44:20.250458 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:20.250363    2578 scope.go:117] "RemoveContainer" containerID="7d5034e1aaa87a995ce66d41af30f9535595d7989202b60b43a5437568d9bc96"
Jul 31 15:44:20.259575 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:20.259467    2578 scope.go:117] "RemoveContainer" containerID="26892e3d48e69185101986f555e67c468141a34e500ed6b178317918e614c40a"
Jul 31 15:44:20.266646 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:20.266625    2578 scope.go:117] "RemoveContainer" containerID="7d5034e1aaa87a995ce66d41af30f9535595d7989202b60b43a5437568d9bc96"
Jul 31 15:44:20.266880 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:44:20.266858    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d5034e1aaa87a995ce66d41af30f9535595d7989202b60b43a5437568d9bc96\": container with ID starting with 7d5034e1aaa87a995ce66d41af30f9535595d7989202b60b43a5437568d9bc96 not found: ID does not exist" containerID="7d5034e1aaa87a995ce66d41af30f9535595d7989202b60b43a5437568d9bc96"
Jul 31 15:44:20.266963 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:20.266886    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d5034e1aaa87a995ce66d41af30f9535595d7989202b60b43a5437568d9bc96"} err="failed to get container status \"7d5034e1aaa87a995ce66d41af30f9535595d7989202b60b43a5437568d9bc96\": rpc error: code = NotFound desc = could not find container \"7d5034e1aaa87a995ce66d41af30f9535595d7989202b60b43a5437568d9bc96\": container with ID starting with 7d5034e1aaa87a995ce66d41af30f9535595d7989202b60b43a5437568d9bc96 not found: ID does not exist"
Jul 31 15:44:20.266963 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:20.266903    2578 scope.go:117] "RemoveContainer" containerID="26892e3d48e69185101986f555e67c468141a34e500ed6b178317918e614c40a"
Jul 31 15:44:20.267106 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:44:20.267087    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26892e3d48e69185101986f555e67c468141a34e500ed6b178317918e614c40a\": container with ID starting with 26892e3d48e69185101986f555e67c468141a34e500ed6b178317918e614c40a not found: ID does not exist" containerID="26892e3d48e69185101986f555e67c468141a34e500ed6b178317918e614c40a"
Jul 31 15:44:20.267168 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:20.267109    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26892e3d48e69185101986f555e67c468141a34e500ed6b178317918e614c40a"} err="failed to get container status \"26892e3d48e69185101986f555e67c468141a34e500ed6b178317918e614c40a\": rpc error: code = NotFound desc = could not find container \"26892e3d48e69185101986f555e67c468141a34e500ed6b178317918e614c40a\": container with ID starting with 26892e3d48e69185101986f555e67c468141a34e500ed6b178317918e614c40a not found: ID does not exist"
Jul 31 15:44:20.274871 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:20.274848    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"]
Jul 31 15:44:20.281382 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:20.281343    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7fb9d5c559-scwzx"]
Jul 31 15:44:20.489285 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:20.489236    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" path="/var/lib/kubelet/pods/483378f4-dcc6-419b-932e-fc655a7e5ec0/volumes"
Jul 31 15:44:28.503081 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:28.503032    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" podUID="75261747-04ae-4586-8d24-0757152500d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 31 15:44:31.231538 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.231494    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"]
Jul 31 15:44:31.232077 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.231896    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerName="storage-initializer"
Jul 31 15:44:31.232077 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.231911    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerName="storage-initializer"
Jul 31 15:44:31.232077 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.231932    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerName="main"
Jul 31 15:44:31.232077 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.231941    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerName="main"
Jul 31 15:44:31.232077 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.232023    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="483378f4-dcc6-419b-932e-fc655a7e5ec0" containerName="main"
Jul 31 15:44:31.237200 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.237174    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.240178 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.240147    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 31 15:44:31.240313 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.240205    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:44:31.240313 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.240251    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 31 15:44:31.240313 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.240269    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Jul 31 15:44:31.240531 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.240408    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-m8lw2\""
Jul 31 15:44:31.241389 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.241369    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 31 15:44:31.252642 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.252615    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"]
Jul 31 15:44:31.294689 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.294657    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-model-cache\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.294843 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.294692    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-tmp-dir\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.294843 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.294716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-home\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.294843 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.294731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-dshm\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.294843 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.294813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0a9f1d8-6529-4336-82f1-306da26bb784-cabundle-cert\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.295012 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.294861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l4p2\" (UniqueName: \"kubernetes.io/projected/e0a9f1d8-6529-4336-82f1-306da26bb784-kube-api-access-6l4p2\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.295012 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.294922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-kserve-provision-location\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.295012 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.294948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a9f1d8-6529-4336-82f1-306da26bb784-tls-certs\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.395933 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.395888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0a9f1d8-6529-4336-82f1-306da26bb784-cabundle-cert\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.396151 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.395949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l4p2\" (UniqueName: \"kubernetes.io/projected/e0a9f1d8-6529-4336-82f1-306da26bb784-kube-api-access-6l4p2\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.396151 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.395993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-kserve-provision-location\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.396151 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.396017    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a9f1d8-6529-4336-82f1-306da26bb784-tls-certs\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.396151 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.396068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-model-cache\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.396151 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.396091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-tmp-dir\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.396151 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.396125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-home\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.396151 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.396147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-dshm\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.396535 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.396474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-tmp-dir\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.396595 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.396574    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-home\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.396662 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.396590    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-kserve-provision-location\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.396761 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.396739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-model-cache\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.396801 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.396766    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0a9f1d8-6529-4336-82f1-306da26bb784-cabundle-cert\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.398621 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.398595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-dshm\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.398793 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.398774    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a9f1d8-6529-4336-82f1-306da26bb784-tls-certs\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.406166 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.406140    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l4p2\" (UniqueName: \"kubernetes.io/projected/e0a9f1d8-6529-4336-82f1-306da26bb784-kube-api-access-6l4p2\") pod \"prestop-hook-test-kserve-56fc49c9b5-9m8sc\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.548116 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.548014    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:31.691309 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:31.691274    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"]
Jul 31 15:44:31.695026 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:44:31.694999    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a9f1d8_6529_4336_82f1_306da26bb784.slice/crio-e887d69aea0c095a11cd2bd24b5f2f6e22265bad9fb12cb06d88cffc5c9a2302 WatchSource:0}: Error finding container e887d69aea0c095a11cd2bd24b5f2f6e22265bad9fb12cb06d88cffc5c9a2302: Status 404 returned error can't find the container with id e887d69aea0c095a11cd2bd24b5f2f6e22265bad9fb12cb06d88cffc5c9a2302
Jul 31 15:44:32.307536 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:32.307492    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" event={"ID":"e0a9f1d8-6529-4336-82f1-306da26bb784","Type":"ContainerStarted","Data":"cb9e5a4e7eaa6f1a5ad0fb739b1f20327747942b28ca15db31aa6da930785fbd"}
Jul 31 15:44:32.308114 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:32.308085    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" event={"ID":"e0a9f1d8-6529-4336-82f1-306da26bb784","Type":"ContainerStarted","Data":"e887d69aea0c095a11cd2bd24b5f2f6e22265bad9fb12cb06d88cffc5c9a2302"}
Jul 31 15:44:34.316803 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:34.316768    2578 generic.go:356] "Generic (PLEG): container finished" podID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerID="cb9e5a4e7eaa6f1a5ad0fb739b1f20327747942b28ca15db31aa6da930785fbd" exitCode=0
Jul 31 15:44:34.317202 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:34.316844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" event={"ID":"e0a9f1d8-6529-4336-82f1-306da26bb784","Type":"ContainerDied","Data":"cb9e5a4e7eaa6f1a5ad0fb739b1f20327747942b28ca15db31aa6da930785fbd"}
Jul 31 15:44:35.323487 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:35.323445    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" event={"ID":"e0a9f1d8-6529-4336-82f1-306da26bb784","Type":"ContainerStarted","Data":"ca7755b9350cedd2596524c2006be3cbc7486911529d3b12ba59babab0c427a9"}
Jul 31 15:44:35.348575 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:35.348504    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" podStartSLOduration=2.011057652 podStartE2EDuration="4.348481123s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.337423471s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:44:35.346085757 +0000 UTC m=+3373.355297628" watchObservedRunningTime="2026-07-31 15:44:35.348481123 +0000 UTC m=+3373.357692993"
Jul 31 15:44:35.548330 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:35.548282    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:38.503446 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:38.503406    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" podUID="75261747-04ae-4586-8d24-0757152500d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 31 15:44:41.548784 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:41.548728    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:44:41.550425 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:41.550390    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 31 15:44:48.503199 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:48.503148    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" podUID="75261747-04ae-4586-8d24-0757152500d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 31 15:44:51.549004 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:51.548953    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 31 15:44:58.503049 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:44:58.502985    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" podUID="75261747-04ae-4586-8d24-0757152500d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 31 15:45:01.549583 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:01.549533    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 31 15:45:08.512697 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:08.512662    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:45:08.520865 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:08.520829    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:45:11.548921 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:11.548874    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 31 15:45:21.549580 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:21.549476    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 31 15:45:27.801691 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:27.801637    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"]
Jul 31 15:45:31.548820 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:31.548762    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 31 15:45:41.549508 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:41.549458    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 31 15:45:42.824184 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:42.824139    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" podUID="75261747-04ae-4586-8d24-0757152500d9" containerName="main" containerID="cri-o://fa13d677375c4e1d176521893978c3c304882d8c0725da27aa5b1e8564665e25" gracePeriod=45
Jul 31 15:45:43.558434 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.558397    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:45:43.615860 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.615771    2578 generic.go:356] "Generic (PLEG): container finished" podID="75261747-04ae-4586-8d24-0757152500d9" containerID="fa13d677375c4e1d176521893978c3c304882d8c0725da27aa5b1e8564665e25" exitCode=0
Jul 31 15:45:43.615860 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.615817    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" event={"ID":"75261747-04ae-4586-8d24-0757152500d9","Type":"ContainerDied","Data":"fa13d677375c4e1d176521893978c3c304882d8c0725da27aa5b1e8564665e25"}
Jul 31 15:45:43.615860 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.615850    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"
Jul 31 15:45:43.616098 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.615870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk" event={"ID":"75261747-04ae-4586-8d24-0757152500d9","Type":"ContainerDied","Data":"15551c29d3cccbfa628fa8f15c81091ad7ab27f39b8923a1b70f51a6c2f86061"}
Jul 31 15:45:43.616098 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.615900    2578 scope.go:117] "RemoveContainer" containerID="fa13d677375c4e1d176521893978c3c304882d8c0725da27aa5b1e8564665e25"
Jul 31 15:45:43.623884 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.623860    2578 scope.go:117] "RemoveContainer" containerID="72ed07c48a1e51dba7932bd849a5e18a57e29bb0975de6cfc3a7612f6b58f081"
Jul 31 15:45:43.632457 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.632434    2578 scope.go:117] "RemoveContainer" containerID="fa13d677375c4e1d176521893978c3c304882d8c0725da27aa5b1e8564665e25"
Jul 31 15:45:43.632721 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:45:43.632701    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa13d677375c4e1d176521893978c3c304882d8c0725da27aa5b1e8564665e25\": container with ID starting with fa13d677375c4e1d176521893978c3c304882d8c0725da27aa5b1e8564665e25 not found: ID does not exist" containerID="fa13d677375c4e1d176521893978c3c304882d8c0725da27aa5b1e8564665e25"
Jul 31 15:45:43.632785 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.632731    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa13d677375c4e1d176521893978c3c304882d8c0725da27aa5b1e8564665e25"} err="failed to get container status \"fa13d677375c4e1d176521893978c3c304882d8c0725da27aa5b1e8564665e25\": rpc error: code = NotFound desc = could not find container \"fa13d677375c4e1d176521893978c3c304882d8c0725da27aa5b1e8564665e25\": container with ID starting with fa13d677375c4e1d176521893978c3c304882d8c0725da27aa5b1e8564665e25 not found: ID does not exist"
Jul 31 15:45:43.632785 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.632750    2578 scope.go:117] "RemoveContainer" containerID="72ed07c48a1e51dba7932bd849a5e18a57e29bb0975de6cfc3a7612f6b58f081"
Jul 31 15:45:43.632999 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:45:43.632975    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ed07c48a1e51dba7932bd849a5e18a57e29bb0975de6cfc3a7612f6b58f081\": container with ID starting with 72ed07c48a1e51dba7932bd849a5e18a57e29bb0975de6cfc3a7612f6b58f081 not found: ID does not exist" containerID="72ed07c48a1e51dba7932bd849a5e18a57e29bb0975de6cfc3a7612f6b58f081"
Jul 31 15:45:43.633038 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.633006    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ed07c48a1e51dba7932bd849a5e18a57e29bb0975de6cfc3a7612f6b58f081"} err="failed to get container status \"72ed07c48a1e51dba7932bd849a5e18a57e29bb0975de6cfc3a7612f6b58f081\": rpc error: code = NotFound desc = could not find container \"72ed07c48a1e51dba7932bd849a5e18a57e29bb0975de6cfc3a7612f6b58f081\": container with ID starting with 72ed07c48a1e51dba7932bd849a5e18a57e29bb0975de6cfc3a7612f6b58f081 not found: ID does not exist"
Jul 31 15:45:43.636254 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.636239    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-kserve-provision-location\") pod \"75261747-04ae-4586-8d24-0757152500d9\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") "
Jul 31 15:45:43.636304 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.636282    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn54m\" (UniqueName: \"kubernetes.io/projected/75261747-04ae-4586-8d24-0757152500d9-kube-api-access-dn54m\") pod \"75261747-04ae-4586-8d24-0757152500d9\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") "
Jul 31 15:45:43.636367 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.636336    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-tmp-dir\") pod \"75261747-04ae-4586-8d24-0757152500d9\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") "
Jul 31 15:45:43.636482 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.636401    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-model-cache\") pod \"75261747-04ae-4586-8d24-0757152500d9\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") "
Jul 31 15:45:43.636482 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.636438    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-home\") pod \"75261747-04ae-4586-8d24-0757152500d9\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") "
Jul 31 15:45:43.636482 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.636460    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75261747-04ae-4586-8d24-0757152500d9-cabundle-cert\") pod \"75261747-04ae-4586-8d24-0757152500d9\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") "
Jul 31 15:45:43.636650 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.636510    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-dshm\") pod \"75261747-04ae-4586-8d24-0757152500d9\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") "
Jul 31 15:45:43.636650 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.636549    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75261747-04ae-4586-8d24-0757152500d9-tls-certs\") pod \"75261747-04ae-4586-8d24-0757152500d9\" (UID: \"75261747-04ae-4586-8d24-0757152500d9\") "
Jul 31 15:45:43.636762 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.636665    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "75261747-04ae-4586-8d24-0757152500d9" (UID: "75261747-04ae-4586-8d24-0757152500d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:45:43.636824 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.636766    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:45:43.637132 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.637092    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-home" (OuterVolumeSpecName: "home") pod "75261747-04ae-4586-8d24-0757152500d9" (UID: "75261747-04ae-4586-8d24-0757152500d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:45:43.637132 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.637110    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75261747-04ae-4586-8d24-0757152500d9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "75261747-04ae-4586-8d24-0757152500d9" (UID: "75261747-04ae-4586-8d24-0757152500d9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:45:43.639096 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.639070    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75261747-04ae-4586-8d24-0757152500d9-kube-api-access-dn54m" (OuterVolumeSpecName: "kube-api-access-dn54m") pod "75261747-04ae-4586-8d24-0757152500d9" (UID: "75261747-04ae-4586-8d24-0757152500d9"). InnerVolumeSpecName "kube-api-access-dn54m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:45:43.639220 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.639103    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-dshm" (OuterVolumeSpecName: "dshm") pod "75261747-04ae-4586-8d24-0757152500d9" (UID: "75261747-04ae-4586-8d24-0757152500d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:45:43.639421 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.639393    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75261747-04ae-4586-8d24-0757152500d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75261747-04ae-4586-8d24-0757152500d9" (UID: "75261747-04ae-4586-8d24-0757152500d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:45:43.650106 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.650084    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75261747-04ae-4586-8d24-0757152500d9" (UID: "75261747-04ae-4586-8d24-0757152500d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:45:43.659183 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.659155    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75261747-04ae-4586-8d24-0757152500d9" (UID: "75261747-04ae-4586-8d24-0757152500d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:45:43.737965 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.737926    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:45:43.737965 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.737960    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75261747-04ae-4586-8d24-0757152500d9-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:45:43.737965 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.737971    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:45:43.738196 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.737979    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75261747-04ae-4586-8d24-0757152500d9-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:45:43.738196 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.737988    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:45:43.738196 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.737996    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn54m\" (UniqueName: \"kubernetes.io/projected/75261747-04ae-4586-8d24-0757152500d9-kube-api-access-dn54m\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:45:43.738196 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.738005    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75261747-04ae-4586-8d24-0757152500d9-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:45:43.940563 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.940530    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"]
Jul 31 15:45:43.946273 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:43.946242    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-6b6bdbb64-xthqk"]
Jul 31 15:45:44.490479 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:44.490437    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75261747-04ae-4586-8d24-0757152500d9" path="/var/lib/kubelet/pods/75261747-04ae-4586-8d24-0757152500d9/volumes"
Jul 31 15:45:48.014545 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.014509    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"]
Jul 31 15:45:48.015006 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.014805    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75261747-04ae-4586-8d24-0757152500d9" containerName="main"
Jul 31 15:45:48.015006 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.014816    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="75261747-04ae-4586-8d24-0757152500d9" containerName="main"
Jul 31 15:45:48.015006 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.014830    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75261747-04ae-4586-8d24-0757152500d9" containerName="storage-initializer"
Jul 31 15:45:48.015006 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.014836    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="75261747-04ae-4586-8d24-0757152500d9" containerName="storage-initializer"
Jul 31 15:45:48.015006 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.014885    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="75261747-04ae-4586-8d24-0757152500d9" containerName="main"
Jul 31 15:45:48.019057 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.019031    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.022003 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.021980    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Jul 31 15:45:48.022145 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.022114    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-fd6c6\""
Jul 31 15:45:48.022228 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.022150    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Jul 31 15:45:48.022403 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.022385    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Jul 31 15:45:48.032710 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.032671    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"]
Jul 31 15:45:48.175719 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.175675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-credential-socket\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.175719 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.175720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797af66c-d587-4875-a577-ea9c26502172-istio-podinfo\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.175939 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.175745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-workload-socket\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.175939 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.175873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-istio-envoy\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.176014 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.175932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797af66c-d587-4875-a577-ea9c26502172-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.176014 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.175971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797af66c-d587-4875-a577-ea9c26502172-istio-token\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.176014 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.175998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-istio-data\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.176128 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.176026    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbtjl\" (UniqueName: \"kubernetes.io/projected/797af66c-d587-4875-a577-ea9c26502172-kube-api-access-gbtjl\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.176128 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.176079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-workload-certs\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.277398 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.277292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-workload-socket\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.277398 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.277385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-istio-envoy\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.277607 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.277423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797af66c-d587-4875-a577-ea9c26502172-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.277607 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.277460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797af66c-d587-4875-a577-ea9c26502172-istio-token\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.277607 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.277486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-istio-data\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.277607 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.277517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbtjl\" (UniqueName: \"kubernetes.io/projected/797af66c-d587-4875-a577-ea9c26502172-kube-api-access-gbtjl\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.277812 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.277698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-workload-certs\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.277812 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.277772    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-workload-socket\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.277812 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.277788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-credential-socket\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.277957 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.277817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797af66c-d587-4875-a577-ea9c26502172-istio-podinfo\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.278083 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.278060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-workload-certs\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.278143 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.278117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-credential-socket\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.278198 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.278183    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797af66c-d587-4875-a577-ea9c26502172-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.278237 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.278216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-istio-data\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.279765 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.279744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-istio-envoy\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.280337 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.280308    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797af66c-d587-4875-a577-ea9c26502172-istio-podinfo\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.286461 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.286433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbtjl\" (UniqueName: \"kubernetes.io/projected/797af66c-d587-4875-a577-ea9c26502172-kube-api-access-gbtjl\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.286623 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.286601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797af66c-d587-4875-a577-ea9c26502172-istio-token\") pod \"router-gateway-2-openshift-default-69f558fd5b-k6t9r\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.330230 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.330190    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:48.463092 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.463055    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"]
Jul 31 15:45:48.466128 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:45:48.466102    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797af66c_d587_4875_a577_ea9c26502172.slice/crio-649715d4e9e1b540bcbb30646a8daf6d282e9f93942c5b2e685cd4b3d8efbb7c WatchSource:0}: Error finding container 649715d4e9e1b540bcbb30646a8daf6d282e9f93942c5b2e685cd4b3d8efbb7c: Status 404 returned error can't find the container with id 649715d4e9e1b540bcbb30646a8daf6d282e9f93942c5b2e685cd4b3d8efbb7c
Jul 31 15:45:48.468691 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.468652    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:45:48.468781 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.468740    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:45:48.468822 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.468778    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:45:48.640724 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.640552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r" event={"ID":"797af66c-d587-4875-a577-ea9c26502172","Type":"ContainerStarted","Data":"78f03661af48b2028e55abe697c8c6fb51afe9e1f47d012e98517b97479ad0fb"}
Jul 31 15:45:48.640724 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.640638    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r" event={"ID":"797af66c-d587-4875-a577-ea9c26502172","Type":"ContainerStarted","Data":"649715d4e9e1b540bcbb30646a8daf6d282e9f93942c5b2e685cd4b3d8efbb7c"}
Jul 31 15:45:48.663074 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:48.663002    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r" podStartSLOduration=1.662981123 podStartE2EDuration="1.662981123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:45:48.662731407 +0000 UTC m=+3446.671943277" watchObservedRunningTime="2026-07-31 15:45:48.662981123 +0000 UTC m=+3446.672192993"
Jul 31 15:45:49.330658 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:49.330619    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:49.335601 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:49.335576    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:49.645373 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:49.645255    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:49.646140 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:49.646113    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:45:51.548985 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:51.548933    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 31 15:45:59.690725 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:59.690682    2578 generic.go:356] "Generic (PLEG): container finished" podID="797af66c-d587-4875-a577-ea9c26502172" containerID="78f03661af48b2028e55abe697c8c6fb51afe9e1f47d012e98517b97479ad0fb" exitCode=0
Jul 31 15:45:59.691144 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:59.690756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r" event={"ID":"797af66c-d587-4875-a577-ea9c26502172","Type":"ContainerDied","Data":"78f03661af48b2028e55abe697c8c6fb51afe9e1f47d012e98517b97479ad0fb"}
Jul 31 15:45:59.691187 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:59.691126    2578 scope.go:117] "RemoveContainer" containerID="78f03661af48b2028e55abe697c8c6fb51afe9e1f47d012e98517b97479ad0fb"
Jul 31 15:45:59.691999 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:59.691969    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:45:59.692110 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:59.692043    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:45:59.692110 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:45:59.692086    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 31 15:46:00.696747 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:00.696706    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r" event={"ID":"797af66c-d587-4875-a577-ea9c26502172","Type":"ContainerStarted","Data":"2b56d0343b3f172bd742bf8ac58c800c4692e413a33ab38a48c0c0f734e733fd"}
Jul 31 15:46:01.330784 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:01.330741    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:46:01.331961 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:01.331921    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r" podUID="797af66c-d587-4875-a577-ea9c26502172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Jul 31 15:46:01.548839 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:01.548788    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 31 15:46:02.330903 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:02.330842    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r" podUID="797af66c-d587-4875-a577-ea9c26502172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Jul 31 15:46:02.888548 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:02.888510    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"]
Jul 31 15:46:02.891966 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:02.891943    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:02.894886 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:02.894850    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:46:02.895032 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:02.894866    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 31 15:46:02.895032 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:02.895018    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Jul 31 15:46:02.895156 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:02.895137    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-klgcq\""
Jul 31 15:46:02.904940 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:02.904910    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"]
Jul 31 15:46:03.008011 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.007968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.008011 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.008017    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-home\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.008276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.008034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.008276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.008060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2985938c-c483-41a4-acfe-9914ef2641df-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.008276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.008122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6bmr\" (UniqueName: \"kubernetes.io/projected/2985938c-c483-41a4-acfe-9914ef2641df-kube-api-access-d6bmr\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.008276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.008176    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2985938c-c483-41a4-acfe-9914ef2641df-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.008276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.008199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.008276 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.008217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.108799 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.108760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.108799 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.108804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-home\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.109032 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.108855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.109032 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.108884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2985938c-c483-41a4-acfe-9914ef2641df-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.109032 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.108955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6bmr\" (UniqueName: \"kubernetes.io/projected/2985938c-c483-41a4-acfe-9914ef2641df-kube-api-access-d6bmr\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.109032 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.108989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2985938c-c483-41a4-acfe-9914ef2641df-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.109032 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.109014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.109320 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.109039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.109320 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.109156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.109320 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.109229    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-home\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.109517 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.109414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.109517 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.109498    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.109815 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.109795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2985938c-c483-41a4-acfe-9914ef2641df-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.111547 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.111526    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2985938c-c483-41a4-acfe-9914ef2641df-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.111668 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.111650    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.118745 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.118719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6bmr\" (UniqueName: \"kubernetes.io/projected/2985938c-c483-41a4-acfe-9914ef2641df-kube-api-access-d6bmr\") pod \"router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.202931 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.202823    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:03.331270 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.331219    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r" podUID="797af66c-d587-4875-a577-ea9c26502172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Jul 31 15:46:03.412077 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.412034    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"]
Jul 31 15:46:03.415692 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:46:03.415659    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2985938c_c483_41a4_acfe_9914ef2641df.slice/crio-e32bed88e71dc4c214e0e022c798a4dce08a4e4b362547ecd354ec040cb174bd WatchSource:0}: Error finding container e32bed88e71dc4c214e0e022c798a4dce08a4e4b362547ecd354ec040cb174bd: Status 404 returned error can't find the container with id e32bed88e71dc4c214e0e022c798a4dce08a4e4b362547ecd354ec040cb174bd
Jul 31 15:46:03.711736 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.711648    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" event={"ID":"2985938c-c483-41a4-acfe-9914ef2641df","Type":"ContainerStarted","Data":"b9c5368dfdc2796e6dd5d71ec15c2ed3676c5f9c69fbade57a0c7c3df023b104"}
Jul 31 15:46:03.711736 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:03.711732    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" event={"ID":"2985938c-c483-41a4-acfe-9914ef2641df","Type":"ContainerStarted","Data":"e32bed88e71dc4c214e0e022c798a4dce08a4e4b362547ecd354ec040cb174bd"}
Jul 31 15:46:04.330686 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:04.330629    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r" podUID="797af66c-d587-4875-a577-ea9c26502172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Jul 31 15:46:04.646260 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:04.646149    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:46:05.330655 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:05.330612    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r" podUID="797af66c-d587-4875-a577-ea9c26502172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Jul 31 15:46:06.335572 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:06.335539    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:46:06.336773 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:06.336747    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:46:06.726541 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:06.726505    2578 generic.go:356] "Generic (PLEG): container finished" podID="2985938c-c483-41a4-acfe-9914ef2641df" containerID="b9c5368dfdc2796e6dd5d71ec15c2ed3676c5f9c69fbade57a0c7c3df023b104" exitCode=0
Jul 31 15:46:06.726711 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:06.726575    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" event={"ID":"2985938c-c483-41a4-acfe-9914ef2641df","Type":"ContainerDied","Data":"b9c5368dfdc2796e6dd5d71ec15c2ed3676c5f9c69fbade57a0c7c3df023b104"}
Jul 31 15:46:07.733838 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:07.733796    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" event={"ID":"2985938c-c483-41a4-acfe-9914ef2641df","Type":"ContainerStarted","Data":"f269060bfefff5ac5e634f2bd56260cdf70910ded5740def6346b59d6ef522cd"}
Jul 31 15:46:07.758769 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:07.758717    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" podStartSLOduration=3.123491911 podStartE2EDuration="5.758703164s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.635211253s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:46:07.757634078 +0000 UTC m=+3465.766845949" watchObservedRunningTime="2026-07-31 15:46:07.758703164 +0000 UTC m=+3465.767915033"
Jul 31 15:46:11.549026 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:11.548965    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 31 15:46:13.203265 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:13.203208    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:13.203895 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:13.203313    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:46:13.205105 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:13.205061    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" podUID="2985938c-c483-41a4-acfe-9914ef2641df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 31 15:46:21.549608 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:21.549554    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 31 15:46:23.203749 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:23.203639    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" podUID="2985938c-c483-41a4-acfe-9914ef2641df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 31 15:46:31.558874 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:31.558837    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:46:31.567165 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:31.567130    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:46:33.204289 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:33.204240    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" podUID="2985938c-c483-41a4-acfe-9914ef2641df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 31 15:46:34.460924 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:34.460880    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"]
Jul 31 15:46:43.203544 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:43.203499    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" podUID="2985938c-c483-41a4-acfe-9914ef2641df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 31 15:46:49.482605 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:49.482512    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerName="main" containerID="cri-o://ca7755b9350cedd2596524c2006be3cbc7486911529d3b12ba59babab0c427a9" gracePeriod=45
Jul 31 15:46:50.234633 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.234608    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:46:50.339001 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.338915    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6l4p2\" (UniqueName: \"kubernetes.io/projected/e0a9f1d8-6529-4336-82f1-306da26bb784-kube-api-access-6l4p2\") pod \"e0a9f1d8-6529-4336-82f1-306da26bb784\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") "
Jul 31 15:46:50.339001 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.338956    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a9f1d8-6529-4336-82f1-306da26bb784-tls-certs\") pod \"e0a9f1d8-6529-4336-82f1-306da26bb784\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") "
Jul 31 15:46:50.339001 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.338992    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-kserve-provision-location\") pod \"e0a9f1d8-6529-4336-82f1-306da26bb784\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") "
Jul 31 15:46:50.339299 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.339014    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-home\") pod \"e0a9f1d8-6529-4336-82f1-306da26bb784\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") "
Jul 31 15:46:50.339299 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.339047    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-model-cache\") pod \"e0a9f1d8-6529-4336-82f1-306da26bb784\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") "
Jul 31 15:46:50.339299 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.339076    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-dshm\") pod \"e0a9f1d8-6529-4336-82f1-306da26bb784\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") "
Jul 31 15:46:50.339299 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.339123    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-tmp-dir\") pod \"e0a9f1d8-6529-4336-82f1-306da26bb784\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") "
Jul 31 15:46:50.339299 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.339146    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0a9f1d8-6529-4336-82f1-306da26bb784-cabundle-cert\") pod \"e0a9f1d8-6529-4336-82f1-306da26bb784\" (UID: \"e0a9f1d8-6529-4336-82f1-306da26bb784\") "
Jul 31 15:46:50.339599 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.339474    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0a9f1d8-6529-4336-82f1-306da26bb784" (UID: "e0a9f1d8-6529-4336-82f1-306da26bb784"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:46:50.339720 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.339691    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a9f1d8-6529-4336-82f1-306da26bb784-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e0a9f1d8-6529-4336-82f1-306da26bb784" (UID: "e0a9f1d8-6529-4336-82f1-306da26bb784"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:46:50.340397 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.340338    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-home" (OuterVolumeSpecName: "home") pod "e0a9f1d8-6529-4336-82f1-306da26bb784" (UID: "e0a9f1d8-6529-4336-82f1-306da26bb784"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:46:50.341847 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.341821    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0a9f1d8-6529-4336-82f1-306da26bb784-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0a9f1d8-6529-4336-82f1-306da26bb784" (UID: "e0a9f1d8-6529-4336-82f1-306da26bb784"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:46:50.341847 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.341830    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a9f1d8-6529-4336-82f1-306da26bb784-kube-api-access-6l4p2" (OuterVolumeSpecName: "kube-api-access-6l4p2") pod "e0a9f1d8-6529-4336-82f1-306da26bb784" (UID: "e0a9f1d8-6529-4336-82f1-306da26bb784"). InnerVolumeSpecName "kube-api-access-6l4p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:46:50.342055 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.342029    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-dshm" (OuterVolumeSpecName: "dshm") pod "e0a9f1d8-6529-4336-82f1-306da26bb784" (UID: "e0a9f1d8-6529-4336-82f1-306da26bb784"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:46:50.352702 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.352669    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0a9f1d8-6529-4336-82f1-306da26bb784" (UID: "e0a9f1d8-6529-4336-82f1-306da26bb784"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:46:50.362283 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.362257    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0a9f1d8-6529-4336-82f1-306da26bb784" (UID: "e0a9f1d8-6529-4336-82f1-306da26bb784"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:46:50.439958 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.439912    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:46:50.439958 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.439959    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0a9f1d8-6529-4336-82f1-306da26bb784-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:46:50.440177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.439976    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l4p2\" (UniqueName: \"kubernetes.io/projected/e0a9f1d8-6529-4336-82f1-306da26bb784-kube-api-access-6l4p2\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:46:50.440177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.439991    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a9f1d8-6529-4336-82f1-306da26bb784-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:46:50.440177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.440006    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:46:50.440177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.440018    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:46:50.440177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.440031    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:46:50.440177 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.440043    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a9f1d8-6529-4336-82f1-306da26bb784-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:46:50.926472 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.926434    2578 generic.go:356] "Generic (PLEG): container finished" podID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerID="ca7755b9350cedd2596524c2006be3cbc7486911529d3b12ba59babab0c427a9" exitCode=0
Jul 31 15:46:50.926902 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.926482    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"
Jul 31 15:46:50.926902 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.926513    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" event={"ID":"e0a9f1d8-6529-4336-82f1-306da26bb784","Type":"ContainerDied","Data":"ca7755b9350cedd2596524c2006be3cbc7486911529d3b12ba59babab0c427a9"}
Jul 31 15:46:50.926902 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.926572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc" event={"ID":"e0a9f1d8-6529-4336-82f1-306da26bb784","Type":"ContainerDied","Data":"e887d69aea0c095a11cd2bd24b5f2f6e22265bad9fb12cb06d88cffc5c9a2302"}
Jul 31 15:46:50.926902 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.926596    2578 scope.go:117] "RemoveContainer" containerID="ca7755b9350cedd2596524c2006be3cbc7486911529d3b12ba59babab0c427a9"
Jul 31 15:46:50.934894 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.934865    2578 scope.go:117] "RemoveContainer" containerID="cb9e5a4e7eaa6f1a5ad0fb739b1f20327747942b28ca15db31aa6da930785fbd"
Jul 31 15:46:50.943588 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.943568    2578 scope.go:117] "RemoveContainer" containerID="ca7755b9350cedd2596524c2006be3cbc7486911529d3b12ba59babab0c427a9"
Jul 31 15:46:50.943862 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:46:50.943842    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca7755b9350cedd2596524c2006be3cbc7486911529d3b12ba59babab0c427a9\": container with ID starting with ca7755b9350cedd2596524c2006be3cbc7486911529d3b12ba59babab0c427a9 not found: ID does not exist" containerID="ca7755b9350cedd2596524c2006be3cbc7486911529d3b12ba59babab0c427a9"
Jul 31 15:46:50.943928 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.943869    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7755b9350cedd2596524c2006be3cbc7486911529d3b12ba59babab0c427a9"} err="failed to get container status \"ca7755b9350cedd2596524c2006be3cbc7486911529d3b12ba59babab0c427a9\": rpc error: code = NotFound desc = could not find container \"ca7755b9350cedd2596524c2006be3cbc7486911529d3b12ba59babab0c427a9\": container with ID starting with ca7755b9350cedd2596524c2006be3cbc7486911529d3b12ba59babab0c427a9 not found: ID does not exist"
Jul 31 15:46:50.943928 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.943888    2578 scope.go:117] "RemoveContainer" containerID="cb9e5a4e7eaa6f1a5ad0fb739b1f20327747942b28ca15db31aa6da930785fbd"
Jul 31 15:46:50.944173 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:46:50.944150    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9e5a4e7eaa6f1a5ad0fb739b1f20327747942b28ca15db31aa6da930785fbd\": container with ID starting with cb9e5a4e7eaa6f1a5ad0fb739b1f20327747942b28ca15db31aa6da930785fbd not found: ID does not exist" containerID="cb9e5a4e7eaa6f1a5ad0fb739b1f20327747942b28ca15db31aa6da930785fbd"
Jul 31 15:46:50.944238 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.944182    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9e5a4e7eaa6f1a5ad0fb739b1f20327747942b28ca15db31aa6da930785fbd"} err="failed to get container status \"cb9e5a4e7eaa6f1a5ad0fb739b1f20327747942b28ca15db31aa6da930785fbd\": rpc error: code = NotFound desc = could not find container \"cb9e5a4e7eaa6f1a5ad0fb739b1f20327747942b28ca15db31aa6da930785fbd\": container with ID starting with cb9e5a4e7eaa6f1a5ad0fb739b1f20327747942b28ca15db31aa6da930785fbd not found: ID does not exist"
Jul 31 15:46:50.947008 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.946984    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"]
Jul 31 15:46:50.952538 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:50.952514    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-9m8sc"]
Jul 31 15:46:52.489359 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:52.489311    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" path="/var/lib/kubelet/pods/e0a9f1d8-6529-4336-82f1-306da26bb784/volumes"
Jul 31 15:46:53.204186 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:46:53.204141    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" podUID="2985938c-c483-41a4-acfe-9914ef2641df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 31 15:47:03.203305 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:47:03.203257    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" podUID="2985938c-c483-41a4-acfe-9914ef2641df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 31 15:47:13.203804 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:47:13.203751    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" podUID="2985938c-c483-41a4-acfe-9914ef2641df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 31 15:47:23.203984 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:47:23.203881    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" podUID="2985938c-c483-41a4-acfe-9914ef2641df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 31 15:47:33.204031 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:47:33.203977    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" podUID="2985938c-c483-41a4-acfe-9914ef2641df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 31 15:47:43.203709 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:47:43.203664    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" podUID="2985938c-c483-41a4-acfe-9914ef2641df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 31 15:47:53.203966 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:47:53.203911    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" podUID="2985938c-c483-41a4-acfe-9914ef2641df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Jul 31 15:48:03.214323 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:03.214282    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:48:03.222803 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:03.222766    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:48:19.913343 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:19.913254    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"]
Jul 31 15:48:19.913768 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:19.913712    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerName="main"
Jul 31 15:48:19.913768 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:19.913732    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerName="main"
Jul 31 15:48:19.913768 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:19.913758    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerName="storage-initializer"
Jul 31 15:48:19.913768 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:19.913768    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerName="storage-initializer"
Jul 31 15:48:19.913917 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:19.913843    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0a9f1d8-6529-4336-82f1-306da26bb784" containerName="main"
Jul 31 15:48:19.916898 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:19.916867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"
Jul 31 15:48:19.920280 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:19.920247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f249c8-4bc7-47f2-ac8c-26247dcb0b77-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n\" (UID: \"43f249c8-4bc7-47f2-ac8c-26247dcb0b77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"
Jul 31 15:48:19.920456 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:19.920327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jppmw\" (UniqueName: \"kubernetes.io/projected/43f249c8-4bc7-47f2-ac8c-26247dcb0b77-kube-api-access-jppmw\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n\" (UID: \"43f249c8-4bc7-47f2-ac8c-26247dcb0b77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"
Jul 31 15:48:19.921414 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:19.921388    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 31 15:48:19.921554 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:19.921423    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Jul 31 15:48:19.921554 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:19.921391    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-jsbfh\""
Jul 31 15:48:19.921554 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:19.921431    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-54b6d\""
Jul 31 15:48:19.921709 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:19.921388    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Jul 31 15:48:19.927622 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:19.927594    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"]
Jul 31 15:48:20.020846 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:20.020803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f249c8-4bc7-47f2-ac8c-26247dcb0b77-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n\" (UID: \"43f249c8-4bc7-47f2-ac8c-26247dcb0b77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"
Jul 31 15:48:20.021036 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:20.020870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jppmw\" (UniqueName: \"kubernetes.io/projected/43f249c8-4bc7-47f2-ac8c-26247dcb0b77-kube-api-access-jppmw\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n\" (UID: \"43f249c8-4bc7-47f2-ac8c-26247dcb0b77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"
Jul 31 15:48:20.023365 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:20.023330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f249c8-4bc7-47f2-ac8c-26247dcb0b77-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n\" (UID: \"43f249c8-4bc7-47f2-ac8c-26247dcb0b77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"
Jul 31 15:48:20.029386 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:20.029333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jppmw\" (UniqueName: \"kubernetes.io/projected/43f249c8-4bc7-47f2-ac8c-26247dcb0b77-kube-api-access-jppmw\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n\" (UID: \"43f249c8-4bc7-47f2-ac8c-26247dcb0b77\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"
Jul 31 15:48:20.228963 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:20.228926    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"
Jul 31 15:48:20.371515 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:20.371319    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"]
Jul 31 15:48:20.374618 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:48:20.374575    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f249c8_4bc7_47f2_ac8c_26247dcb0b77.slice/crio-c5d85ae56898f022db915564c33e122f1edcf16fb329c66a95bef3a932cded48 WatchSource:0}: Error finding container c5d85ae56898f022db915564c33e122f1edcf16fb329c66a95bef3a932cded48: Status 404 returned error can't find the container with id c5d85ae56898f022db915564c33e122f1edcf16fb329c66a95bef3a932cded48
Jul 31 15:48:21.311633 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:21.311593    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n" event={"ID":"43f249c8-4bc7-47f2-ac8c-26247dcb0b77","Type":"ContainerStarted","Data":"a9a8505684e19c261aeace40e0d6539b9130d10462591aacc432487e6df3e30f"}
Jul 31 15:48:21.312085 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:21.311677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n" event={"ID":"43f249c8-4bc7-47f2-ac8c-26247dcb0b77","Type":"ContainerStarted","Data":"c5d85ae56898f022db915564c33e122f1edcf16fb329c66a95bef3a932cded48"}
Jul 31 15:48:21.312085 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:21.311755    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"
Jul 31 15:48:21.329845 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:21.329792    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n" podStartSLOduration=2.329777754 podStartE2EDuration="2.329777754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:48:21.329160342 +0000 UTC m=+3599.338372213" watchObservedRunningTime="2026-07-31 15:48:21.329777754 +0000 UTC m=+3599.338989622"
Jul 31 15:48:42.822450 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:42.822412    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"]
Jul 31 15:48:52.317624 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:52.317587    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"
Jul 31 15:48:53.502571 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.502531    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"]
Jul 31 15:48:53.506280 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.506260    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.516708 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.516664    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"]
Jul 31 15:48:53.616135 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.616088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5r5b\" (UniqueName: \"kubernetes.io/projected/5beba8c1-4060-4e95-8f0f-efce040a380a-kube-api-access-q5r5b\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.616323 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.616145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.616323 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.616249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.616323 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.616294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5beba8c1-4060-4e95-8f0f-efce040a380a-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.616323 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.616315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.616512 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.616342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.717100 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.717056    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5r5b\" (UniqueName: \"kubernetes.io/projected/5beba8c1-4060-4e95-8f0f-efce040a380a-kube-api-access-q5r5b\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.717100 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.717103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.717332 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.717152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.717332 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.717181    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5beba8c1-4060-4e95-8f0f-efce040a380a-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.717332 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.717209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.717332 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.717242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.717739 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.717717    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.717739 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.717724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.717909 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.717774    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.719601 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.719576    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.719892 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.719872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5beba8c1-4060-4e95-8f0f-efce040a380a-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.724967 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.724943    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5r5b\" (UniqueName: \"kubernetes.io/projected/5beba8c1-4060-4e95-8f0f-efce040a380a-kube-api-access-q5r5b\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-2tmcc\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.817678 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.817579    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:53.947980 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:53.947950    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"]
Jul 31 15:48:53.950025 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:48:53.949998    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5beba8c1_4060_4e95_8f0f_efce040a380a.slice/crio-ae94dad36009759dd7cb2e33be1fc45d69b5d698a4551337b8811ec704c85adc WatchSource:0}: Error finding container ae94dad36009759dd7cb2e33be1fc45d69b5d698a4551337b8811ec704c85adc: Status 404 returned error can't find the container with id ae94dad36009759dd7cb2e33be1fc45d69b5d698a4551337b8811ec704c85adc
Jul 31 15:48:54.448916 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:54.448873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc" event={"ID":"5beba8c1-4060-4e95-8f0f-efce040a380a","Type":"ContainerStarted","Data":"e330001a1370c74256364ef5353130af646226bc3b8db4a0aae62d62170293f4"}
Jul 31 15:48:54.449128 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:54.448926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc" event={"ID":"5beba8c1-4060-4e95-8f0f-efce040a380a","Type":"ContainerStarted","Data":"ae94dad36009759dd7cb2e33be1fc45d69b5d698a4551337b8811ec704c85adc"}
Jul 31 15:48:54.470017 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:54.469964    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc" podStartSLOduration=1.469949339 podStartE2EDuration="1.469949339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:48:54.467631358 +0000 UTC m=+3632.476843228" watchObservedRunningTime="2026-07-31 15:48:54.469949339 +0000 UTC m=+3632.479161207"
Jul 31 15:48:54.818730 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:54.818689    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:48:57.845277 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:57.845231    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" podUID="2985938c-c483-41a4-acfe-9914ef2641df" containerName="main" containerID="cri-o://f269060bfefff5ac5e634f2bd56260cdf70910ded5740def6346b59d6ef522cd" gracePeriod=45
Jul 31 15:48:58.466416 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.466381    2578 generic.go:356] "Generic (PLEG): container finished" podID="2985938c-c483-41a4-acfe-9914ef2641df" containerID="f269060bfefff5ac5e634f2bd56260cdf70910ded5740def6346b59d6ef522cd" exitCode=0
Jul 31 15:48:58.466543 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.466393    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" event={"ID":"2985938c-c483-41a4-acfe-9914ef2641df","Type":"ContainerDied","Data":"f269060bfefff5ac5e634f2bd56260cdf70910ded5740def6346b59d6ef522cd"}
Jul 31 15:48:58.584455 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.584424    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:48:58.659678 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.659575    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-kserve-provision-location\") pod \"2985938c-c483-41a4-acfe-9914ef2641df\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") "
Jul 31 15:48:58.659678 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.659621    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-model-cache\") pod \"2985938c-c483-41a4-acfe-9914ef2641df\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") "
Jul 31 15:48:58.659678 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.659655    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2985938c-c483-41a4-acfe-9914ef2641df-cabundle-cert\") pod \"2985938c-c483-41a4-acfe-9914ef2641df\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") "
Jul 31 15:48:58.659970 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.659703    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-home\") pod \"2985938c-c483-41a4-acfe-9914ef2641df\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") "
Jul 31 15:48:58.659970 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.659855    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-tmp-dir\") pod \"2985938c-c483-41a4-acfe-9914ef2641df\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") "
Jul 31 15:48:58.659970 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.659907    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6bmr\" (UniqueName: \"kubernetes.io/projected/2985938c-c483-41a4-acfe-9914ef2641df-kube-api-access-d6bmr\") pod \"2985938c-c483-41a4-acfe-9914ef2641df\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") "
Jul 31 15:48:58.659970 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.659934    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-model-cache" (OuterVolumeSpecName: "model-cache") pod "2985938c-c483-41a4-acfe-9914ef2641df" (UID: "2985938c-c483-41a4-acfe-9914ef2641df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:48:58.659970 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.659955    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-dshm\") pod \"2985938c-c483-41a4-acfe-9914ef2641df\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") "
Jul 31 15:48:58.660236 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.659988    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2985938c-c483-41a4-acfe-9914ef2641df-tls-certs\") pod \"2985938c-c483-41a4-acfe-9914ef2641df\" (UID: \"2985938c-c483-41a4-acfe-9914ef2641df\") "
Jul 31 15:48:58.660236 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.660037    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2985938c-c483-41a4-acfe-9914ef2641df-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2985938c-c483-41a4-acfe-9914ef2641df" (UID: "2985938c-c483-41a4-acfe-9914ef2641df"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:48:58.660440 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.660391    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:48:58.660440 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.660417    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2985938c-c483-41a4-acfe-9914ef2641df-cabundle-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:48:58.660627 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.660465    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-home" (OuterVolumeSpecName: "home") pod "2985938c-c483-41a4-acfe-9914ef2641df" (UID: "2985938c-c483-41a4-acfe-9914ef2641df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:48:58.662333 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.662289    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2985938c-c483-41a4-acfe-9914ef2641df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2985938c-c483-41a4-acfe-9914ef2641df" (UID: "2985938c-c483-41a4-acfe-9914ef2641df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:48:58.662632 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.662609    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-dshm" (OuterVolumeSpecName: "dshm") pod "2985938c-c483-41a4-acfe-9914ef2641df" (UID: "2985938c-c483-41a4-acfe-9914ef2641df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:48:58.662705 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.662652    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2985938c-c483-41a4-acfe-9914ef2641df-kube-api-access-d6bmr" (OuterVolumeSpecName: "kube-api-access-d6bmr") pod "2985938c-c483-41a4-acfe-9914ef2641df" (UID: "2985938c-c483-41a4-acfe-9914ef2641df"). InnerVolumeSpecName "kube-api-access-d6bmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:48:58.676253 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.676218    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2985938c-c483-41a4-acfe-9914ef2641df" (UID: "2985938c-c483-41a4-acfe-9914ef2641df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:48:58.690413 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.690377    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2985938c-c483-41a4-acfe-9914ef2641df" (UID: "2985938c-c483-41a4-acfe-9914ef2641df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:48:58.761372 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.761301    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:48:58.761372 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.761341    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:48:58.761372 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.761376    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6bmr\" (UniqueName: \"kubernetes.io/projected/2985938c-c483-41a4-acfe-9914ef2641df-kube-api-access-d6bmr\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:48:58.761603 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.761387    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:48:58.761603 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.761396    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2985938c-c483-41a4-acfe-9914ef2641df-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:48:58.761603 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:58.761405    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2985938c-c483-41a4-acfe-9914ef2641df-kserve-provision-location\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:48:59.471548 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:59.471512    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"
Jul 31 15:48:59.471937 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:59.471513    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn" event={"ID":"2985938c-c483-41a4-acfe-9914ef2641df","Type":"ContainerDied","Data":"e32bed88e71dc4c214e0e022c798a4dce08a4e4b362547ecd354ec040cb174bd"}
Jul 31 15:48:59.471937 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:59.471661    2578 scope.go:117] "RemoveContainer" containerID="f269060bfefff5ac5e634f2bd56260cdf70910ded5740def6346b59d6ef522cd"
Jul 31 15:48:59.480097 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:59.480076    2578 scope.go:117] "RemoveContainer" containerID="b9c5368dfdc2796e6dd5d71ec15c2ed3676c5f9c69fbade57a0c7c3df023b104"
Jul 31 15:48:59.494679 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:59.494653    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"]
Jul 31 15:48:59.499419 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:48:59.499393    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694fd77cc6-skjrn"]
Jul 31 15:49:00.489795 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:00.489758    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2985938c-c483-41a4-acfe-9914ef2641df" path="/var/lib/kubelet/pods/2985938c-c483-41a4-acfe-9914ef2641df/volumes"
Jul 31 15:49:03.818667 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:03.818576    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:49:03.831110 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:03.831076    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:49:04.500697 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:04.500662    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:49:07.784101 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:07.784055    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"]
Jul 31 15:49:07.784506 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:07.784267    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r" podUID="797af66c-d587-4875-a577-ea9c26502172" containerName="istio-proxy" containerID="cri-o://2b56d0343b3f172bd742bf8ac58c800c4692e413a33ab38a48c0c0f734e733fd" gracePeriod=30
Jul 31 15:49:12.600026 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:12.599993    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"]
Jul 31 15:49:12.609334 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:12.609305    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"]
Jul 31 15:49:12.617912 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:49:12.617870    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:49:12.617912 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:49:12.617912 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:49:12.617912 ip-10-0-133-32 kubenswrapper[2578]:  > containerID="e330001a1370c74256364ef5353130af646226bc3b8db4a0aae62d62170293f4" cmd=["/bin/sleep","15"]
Jul 31 15:49:12.618164 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:12.617934    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc" message="" err=<
Jul 31 15:49:12.618164 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:49:12.618164 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:49:12.618164 ip-10-0-133-32 kubenswrapper[2578]:  >
Jul 31 15:49:12.618164 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:49:12.617952    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:49:12.618164 ip-10-0-133-32 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:49:12.618164 ip-10-0-133-32 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 31 15:49:12.618164 ip-10-0-133-32 kubenswrapper[2578]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc" podUID="5beba8c1-4060-4e95-8f0f-efce040a380a" containerName="main" containerID="cri-o://e330001a1370c74256364ef5353130af646226bc3b8db4a0aae62d62170293f4"
Jul 31 15:49:12.618164 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:12.618001    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc" podUID="5beba8c1-4060-4e95-8f0f-efce040a380a" containerName="main" containerID="cri-o://e330001a1370c74256364ef5353130af646226bc3b8db4a0aae62d62170293f4" gracePeriod=60
Jul 31 15:49:13.022905 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.022879    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:49:13.052806 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.052780    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:49:13.187629 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187530    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797af66c-d587-4875-a577-ea9c26502172-istiod-ca-cert\") pod \"797af66c-d587-4875-a577-ea9c26502172\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") "
Jul 31 15:49:13.187629 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187588    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-home\") pod \"5beba8c1-4060-4e95-8f0f-efce040a380a\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") "
Jul 31 15:49:13.187629 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187620    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-credential-socket\") pod \"797af66c-d587-4875-a577-ea9c26502172\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") "
Jul 31 15:49:13.187917 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187638    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbtjl\" (UniqueName: \"kubernetes.io/projected/797af66c-d587-4875-a577-ea9c26502172-kube-api-access-gbtjl\") pod \"797af66c-d587-4875-a577-ea9c26502172\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") "
Jul 31 15:49:13.187917 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187661    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797af66c-d587-4875-a577-ea9c26502172-istio-podinfo\") pod \"797af66c-d587-4875-a577-ea9c26502172\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") "
Jul 31 15:49:13.187917 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187681    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5r5b\" (UniqueName: \"kubernetes.io/projected/5beba8c1-4060-4e95-8f0f-efce040a380a-kube-api-access-q5r5b\") pod \"5beba8c1-4060-4e95-8f0f-efce040a380a\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") "
Jul 31 15:49:13.187917 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187717    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5beba8c1-4060-4e95-8f0f-efce040a380a-tls-certs\") pod \"5beba8c1-4060-4e95-8f0f-efce040a380a\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") "
Jul 31 15:49:13.187917 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187754    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-istio-data\") pod \"797af66c-d587-4875-a577-ea9c26502172\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") "
Jul 31 15:49:13.187917 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187776    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-dshm\") pod \"5beba8c1-4060-4e95-8f0f-efce040a380a\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") "
Jul 31 15:49:13.187917 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187798    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-tmp-dir\") pod \"5beba8c1-4060-4e95-8f0f-efce040a380a\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") "
Jul 31 15:49:13.187917 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187835    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-workload-certs\") pod \"797af66c-d587-4875-a577-ea9c26502172\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") "
Jul 31 15:49:13.187917 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187864    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797af66c-d587-4875-a577-ea9c26502172-istio-token\") pod \"797af66c-d587-4875-a577-ea9c26502172\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") "
Jul 31 15:49:13.187917 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187869    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-home" (OuterVolumeSpecName: "home") pod "5beba8c1-4060-4e95-8f0f-efce040a380a" (UID: "5beba8c1-4060-4e95-8f0f-efce040a380a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:49:13.187917 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187888    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-workload-socket\") pod \"797af66c-d587-4875-a577-ea9c26502172\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") "
Jul 31 15:49:13.187917 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187896    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "797af66c-d587-4875-a577-ea9c26502172" (UID: "797af66c-d587-4875-a577-ea9c26502172"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:49:13.187917 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187913    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-model-cache\") pod \"5beba8c1-4060-4e95-8f0f-efce040a380a\" (UID: \"5beba8c1-4060-4e95-8f0f-efce040a380a\") "
Jul 31 15:49:13.188738 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187930    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/797af66c-d587-4875-a577-ea9c26502172-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "797af66c-d587-4875-a577-ea9c26502172" (UID: "797af66c-d587-4875-a577-ea9c26502172"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:49:13.188738 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.187943    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-istio-envoy\") pod \"797af66c-d587-4875-a577-ea9c26502172\" (UID: \"797af66c-d587-4875-a577-ea9c26502172\") "
Jul 31 15:49:13.188738 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.188044    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "797af66c-d587-4875-a577-ea9c26502172" (UID: "797af66c-d587-4875-a577-ea9c26502172"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:49:13.188738 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.188071    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-istio-data" (OuterVolumeSpecName: "istio-data") pod "797af66c-d587-4875-a577-ea9c26502172" (UID: "797af66c-d587-4875-a577-ea9c26502172"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:49:13.188738 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.188249    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:49:13.188738 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.188271    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:49:13.188738 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.188286    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797af66c-d587-4875-a577-ea9c26502172-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:49:13.188738 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.188299    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:49:13.188738 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.188311    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:49:13.188738 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.188247    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "797af66c-d587-4875-a577-ea9c26502172" (UID: "797af66c-d587-4875-a577-ea9c26502172"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:49:13.188738 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.188450    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5beba8c1-4060-4e95-8f0f-efce040a380a" (UID: "5beba8c1-4060-4e95-8f0f-efce040a380a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:49:13.188738 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.188672    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-model-cache" (OuterVolumeSpecName: "model-cache") pod "5beba8c1-4060-4e95-8f0f-efce040a380a" (UID: "5beba8c1-4060-4e95-8f0f-efce040a380a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:49:13.190599 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.190568    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "797af66c-d587-4875-a577-ea9c26502172" (UID: "797af66c-d587-4875-a577-ea9c26502172"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:49:13.190732 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.190666    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797af66c-d587-4875-a577-ea9c26502172-kube-api-access-gbtjl" (OuterVolumeSpecName: "kube-api-access-gbtjl") pod "797af66c-d587-4875-a577-ea9c26502172" (UID: "797af66c-d587-4875-a577-ea9c26502172"). InnerVolumeSpecName "kube-api-access-gbtjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:49:13.190732 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.190689    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797af66c-d587-4875-a577-ea9c26502172-istio-token" (OuterVolumeSpecName: "istio-token") pod "797af66c-d587-4875-a577-ea9c26502172" (UID: "797af66c-d587-4875-a577-ea9c26502172"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:49:13.190732 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.190689    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/797af66c-d587-4875-a577-ea9c26502172-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "797af66c-d587-4875-a577-ea9c26502172" (UID: "797af66c-d587-4875-a577-ea9c26502172"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 15:49:13.190888 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.190756    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5beba8c1-4060-4e95-8f0f-efce040a380a-kube-api-access-q5r5b" (OuterVolumeSpecName: "kube-api-access-q5r5b") pod "5beba8c1-4060-4e95-8f0f-efce040a380a" (UID: "5beba8c1-4060-4e95-8f0f-efce040a380a"). InnerVolumeSpecName "kube-api-access-q5r5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:49:13.190888 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.190774    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-dshm" (OuterVolumeSpecName: "dshm") pod "5beba8c1-4060-4e95-8f0f-efce040a380a" (UID: "5beba8c1-4060-4e95-8f0f-efce040a380a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:49:13.191023 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.190997    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5beba8c1-4060-4e95-8f0f-efce040a380a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5beba8c1-4060-4e95-8f0f-efce040a380a" (UID: "5beba8c1-4060-4e95-8f0f-efce040a380a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:49:13.289184 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.289147    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797af66c-d587-4875-a577-ea9c26502172-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:49:13.289184 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.289179    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:49:13.289184 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.289190    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:49:13.289468 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.289199    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797af66c-d587-4875-a577-ea9c26502172-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:49:13.289468 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.289208    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbtjl\" (UniqueName: \"kubernetes.io/projected/797af66c-d587-4875-a577-ea9c26502172-kube-api-access-gbtjl\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:49:13.289468 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.289217    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797af66c-d587-4875-a577-ea9c26502172-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:49:13.289468 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.289226    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5r5b\" (UniqueName: \"kubernetes.io/projected/5beba8c1-4060-4e95-8f0f-efce040a380a-kube-api-access-q5r5b\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:49:13.289468 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.289236    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5beba8c1-4060-4e95-8f0f-efce040a380a-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:49:13.289468 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.289245    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:49:13.289468 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.289253    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5beba8c1-4060-4e95-8f0f-efce040a380a-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:49:13.523587 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.523548    2578 generic.go:356] "Generic (PLEG): container finished" podID="797af66c-d587-4875-a577-ea9c26502172" containerID="2b56d0343b3f172bd742bf8ac58c800c4692e413a33ab38a48c0c0f734e733fd" exitCode=0
Jul 31 15:49:13.523783 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.523613    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"
Jul 31 15:49:13.523783 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.523629    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r" event={"ID":"797af66c-d587-4875-a577-ea9c26502172","Type":"ContainerDied","Data":"2b56d0343b3f172bd742bf8ac58c800c4692e413a33ab38a48c0c0f734e733fd"}
Jul 31 15:49:13.523783 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.523689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r" event={"ID":"797af66c-d587-4875-a577-ea9c26502172","Type":"ContainerDied","Data":"649715d4e9e1b540bcbb30646a8daf6d282e9f93942c5b2e685cd4b3d8efbb7c"}
Jul 31 15:49:13.523783 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.523713    2578 scope.go:117] "RemoveContainer" containerID="2b56d0343b3f172bd742bf8ac58c800c4692e413a33ab38a48c0c0f734e733fd"
Jul 31 15:49:13.525001 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.524983    2578 generic.go:356] "Generic (PLEG): container finished" podID="5beba8c1-4060-4e95-8f0f-efce040a380a" containerID="e330001a1370c74256364ef5353130af646226bc3b8db4a0aae62d62170293f4" exitCode=0
Jul 31 15:49:13.525103 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.525022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc" event={"ID":"5beba8c1-4060-4e95-8f0f-efce040a380a","Type":"ContainerDied","Data":"e330001a1370c74256364ef5353130af646226bc3b8db4a0aae62d62170293f4"}
Jul 31 15:49:13.525103 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.525053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc" event={"ID":"5beba8c1-4060-4e95-8f0f-efce040a380a","Type":"ContainerDied","Data":"ae94dad36009759dd7cb2e33be1fc45d69b5d698a4551337b8811ec704c85adc"}
Jul 31 15:49:13.525103 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.525059    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"
Jul 31 15:49:13.533505 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.533459    2578 scope.go:117] "RemoveContainer" containerID="78f03661af48b2028e55abe697c8c6fb51afe9e1f47d012e98517b97479ad0fb"
Jul 31 15:49:13.542007 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.541990    2578 scope.go:117] "RemoveContainer" containerID="2b56d0343b3f172bd742bf8ac58c800c4692e413a33ab38a48c0c0f734e733fd"
Jul 31 15:49:13.542265 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:49:13.542246    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b56d0343b3f172bd742bf8ac58c800c4692e413a33ab38a48c0c0f734e733fd\": container with ID starting with 2b56d0343b3f172bd742bf8ac58c800c4692e413a33ab38a48c0c0f734e733fd not found: ID does not exist" containerID="2b56d0343b3f172bd742bf8ac58c800c4692e413a33ab38a48c0c0f734e733fd"
Jul 31 15:49:13.542321 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.542273    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b56d0343b3f172bd742bf8ac58c800c4692e413a33ab38a48c0c0f734e733fd"} err="failed to get container status \"2b56d0343b3f172bd742bf8ac58c800c4692e413a33ab38a48c0c0f734e733fd\": rpc error: code = NotFound desc = could not find container \"2b56d0343b3f172bd742bf8ac58c800c4692e413a33ab38a48c0c0f734e733fd\": container with ID starting with 2b56d0343b3f172bd742bf8ac58c800c4692e413a33ab38a48c0c0f734e733fd not found: ID does not exist"
Jul 31 15:49:13.542321 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.542292    2578 scope.go:117] "RemoveContainer" containerID="78f03661af48b2028e55abe697c8c6fb51afe9e1f47d012e98517b97479ad0fb"
Jul 31 15:49:13.542535 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:49:13.542516    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f03661af48b2028e55abe697c8c6fb51afe9e1f47d012e98517b97479ad0fb\": container with ID starting with 78f03661af48b2028e55abe697c8c6fb51afe9e1f47d012e98517b97479ad0fb not found: ID does not exist" containerID="78f03661af48b2028e55abe697c8c6fb51afe9e1f47d012e98517b97479ad0fb"
Jul 31 15:49:13.542579 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.542540    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f03661af48b2028e55abe697c8c6fb51afe9e1f47d012e98517b97479ad0fb"} err="failed to get container status \"78f03661af48b2028e55abe697c8c6fb51afe9e1f47d012e98517b97479ad0fb\": rpc error: code = NotFound desc = could not find container \"78f03661af48b2028e55abe697c8c6fb51afe9e1f47d012e98517b97479ad0fb\": container with ID starting with 78f03661af48b2028e55abe697c8c6fb51afe9e1f47d012e98517b97479ad0fb not found: ID does not exist"
Jul 31 15:49:13.542579 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.542556    2578 scope.go:117] "RemoveContainer" containerID="e330001a1370c74256364ef5353130af646226bc3b8db4a0aae62d62170293f4"
Jul 31 15:49:13.549949 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.549931    2578 scope.go:117] "RemoveContainer" containerID="e330001a1370c74256364ef5353130af646226bc3b8db4a0aae62d62170293f4"
Jul 31 15:49:13.550206 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:49:13.550189    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e330001a1370c74256364ef5353130af646226bc3b8db4a0aae62d62170293f4\": container with ID starting with e330001a1370c74256364ef5353130af646226bc3b8db4a0aae62d62170293f4 not found: ID does not exist" containerID="e330001a1370c74256364ef5353130af646226bc3b8db4a0aae62d62170293f4"
Jul 31 15:49:13.550255 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.550211    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e330001a1370c74256364ef5353130af646226bc3b8db4a0aae62d62170293f4"} err="failed to get container status \"e330001a1370c74256364ef5353130af646226bc3b8db4a0aae62d62170293f4\": rpc error: code = NotFound desc = could not find container \"e330001a1370c74256364ef5353130af646226bc3b8db4a0aae62d62170293f4\": container with ID starting with e330001a1370c74256364ef5353130af646226bc3b8db4a0aae62d62170293f4 not found: ID does not exist"
Jul 31 15:49:13.560052 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.560030    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"]
Jul 31 15:49:13.565724 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.565702    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-69f558fd5b-k6t9r"]
Jul 31 15:49:13.583451 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.583428    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"]
Jul 31 15:49:13.590462 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:13.590438    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-2tmcc"]
Jul 31 15:49:14.489064 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:14.489026    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5beba8c1-4060-4e95-8f0f-efce040a380a" path="/var/lib/kubelet/pods/5beba8c1-4060-4e95-8f0f-efce040a380a/volumes"
Jul 31 15:49:14.489472 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:14.489458    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="797af66c-d587-4875-a577-ea9c26502172" path="/var/lib/kubelet/pods/797af66c-d587-4875-a577-ea9c26502172/volumes"
Jul 31 15:49:22.317482 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:22.317440    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n" podUID="43f249c8-4bc7-47f2-ac8c-26247dcb0b77" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 31 15:49:27.620260 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:27.620189    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n" podUID="43f249c8-4bc7-47f2-ac8c-26247dcb0b77" containerName="main" containerID="cri-o://a9a8505684e19c261aeace40e0d6539b9130d10462591aacc432487e6df3e30f" gracePeriod=45
Jul 31 15:49:27.862449 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:27.862421    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"
Jul 31 15:49:27.892841 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:27.892771    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f249c8-4bc7-47f2-ac8c-26247dcb0b77-tls-certs\") pod \"43f249c8-4bc7-47f2-ac8c-26247dcb0b77\" (UID: \"43f249c8-4bc7-47f2-ac8c-26247dcb0b77\") "
Jul 31 15:49:27.892841 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:27.892834    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jppmw\" (UniqueName: \"kubernetes.io/projected/43f249c8-4bc7-47f2-ac8c-26247dcb0b77-kube-api-access-jppmw\") pod \"43f249c8-4bc7-47f2-ac8c-26247dcb0b77\" (UID: \"43f249c8-4bc7-47f2-ac8c-26247dcb0b77\") "
Jul 31 15:49:27.894981 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:27.894954    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f249c8-4bc7-47f2-ac8c-26247dcb0b77-kube-api-access-jppmw" (OuterVolumeSpecName: "kube-api-access-jppmw") pod "43f249c8-4bc7-47f2-ac8c-26247dcb0b77" (UID: "43f249c8-4bc7-47f2-ac8c-26247dcb0b77"). InnerVolumeSpecName "kube-api-access-jppmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:49:27.895114 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:27.894976    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f249c8-4bc7-47f2-ac8c-26247dcb0b77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43f249c8-4bc7-47f2-ac8c-26247dcb0b77" (UID: "43f249c8-4bc7-47f2-ac8c-26247dcb0b77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:49:27.994313 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:27.994265    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f249c8-4bc7-47f2-ac8c-26247dcb0b77-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:49:27.994313 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:27.994308    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jppmw\" (UniqueName: \"kubernetes.io/projected/43f249c8-4bc7-47f2-ac8c-26247dcb0b77-kube-api-access-jppmw\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:49:28.583726 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:28.583687    2578 generic.go:356] "Generic (PLEG): container finished" podID="43f249c8-4bc7-47f2-ac8c-26247dcb0b77" containerID="a9a8505684e19c261aeace40e0d6539b9130d10462591aacc432487e6df3e30f" exitCode=0
Jul 31 15:49:28.583902 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:28.583737    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"
Jul 31 15:49:28.583902 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:28.583761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n" event={"ID":"43f249c8-4bc7-47f2-ac8c-26247dcb0b77","Type":"ContainerDied","Data":"a9a8505684e19c261aeace40e0d6539b9130d10462591aacc432487e6df3e30f"}
Jul 31 15:49:28.583902 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:28.583813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n" event={"ID":"43f249c8-4bc7-47f2-ac8c-26247dcb0b77","Type":"ContainerDied","Data":"c5d85ae56898f022db915564c33e122f1edcf16fb329c66a95bef3a932cded48"}
Jul 31 15:49:28.583902 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:28.583841    2578 scope.go:117] "RemoveContainer" containerID="a9a8505684e19c261aeace40e0d6539b9130d10462591aacc432487e6df3e30f"
Jul 31 15:49:28.591938 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:28.591911    2578 scope.go:117] "RemoveContainer" containerID="a9a8505684e19c261aeace40e0d6539b9130d10462591aacc432487e6df3e30f"
Jul 31 15:49:28.592184 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:49:28.592166    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9a8505684e19c261aeace40e0d6539b9130d10462591aacc432487e6df3e30f\": container with ID starting with a9a8505684e19c261aeace40e0d6539b9130d10462591aacc432487e6df3e30f not found: ID does not exist" containerID="a9a8505684e19c261aeace40e0d6539b9130d10462591aacc432487e6df3e30f"
Jul 31 15:49:28.592233 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:28.592192    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9a8505684e19c261aeace40e0d6539b9130d10462591aacc432487e6df3e30f"} err="failed to get container status \"a9a8505684e19c261aeace40e0d6539b9130d10462591aacc432487e6df3e30f\": rpc error: code = NotFound desc = could not find container \"a9a8505684e19c261aeace40e0d6539b9130d10462591aacc432487e6df3e30f\": container with ID starting with a9a8505684e19c261aeace40e0d6539b9130d10462591aacc432487e6df3e30f not found: ID does not exist"
Jul 31 15:49:28.601125 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:28.601092    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"]
Jul 31 15:49:28.605009 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:28.604982    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-88c7n"]
Jul 31 15:49:30.491422 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:49:30.491374    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43f249c8-4bc7-47f2-ac8c-26247dcb0b77" path="/var/lib/kubelet/pods/43f249c8-4bc7-47f2-ac8c-26247dcb0b77/volumes"
Jul 31 15:51:50.186596 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.186558    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vkjj8/must-gather-h2klp"]
Jul 31 15:51:50.187010 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.186824    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f249c8-4bc7-47f2-ac8c-26247dcb0b77" containerName="main"
Jul 31 15:51:50.187010 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.186834    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f249c8-4bc7-47f2-ac8c-26247dcb0b77" containerName="main"
Jul 31 15:51:50.187010 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.186848    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5beba8c1-4060-4e95-8f0f-efce040a380a" containerName="main"
Jul 31 15:51:50.187010 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.186853    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5beba8c1-4060-4e95-8f0f-efce040a380a" containerName="main"
Jul 31 15:51:50.187010 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.186859    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="797af66c-d587-4875-a577-ea9c26502172" containerName="istio-proxy"
Jul 31 15:51:50.187010 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.186865    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="797af66c-d587-4875-a577-ea9c26502172" containerName="istio-proxy"
Jul 31 15:51:50.187010 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.186877    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2985938c-c483-41a4-acfe-9914ef2641df" containerName="storage-initializer"
Jul 31 15:51:50.187010 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.186883    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2985938c-c483-41a4-acfe-9914ef2641df" containerName="storage-initializer"
Jul 31 15:51:50.187010 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.186891    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="797af66c-d587-4875-a577-ea9c26502172" containerName="istio-proxy"
Jul 31 15:51:50.187010 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.186896    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="797af66c-d587-4875-a577-ea9c26502172" containerName="istio-proxy"
Jul 31 15:51:50.187010 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.186904    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2985938c-c483-41a4-acfe-9914ef2641df" containerName="main"
Jul 31 15:51:50.187010 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.186911    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2985938c-c483-41a4-acfe-9914ef2641df" containerName="main"
Jul 31 15:51:50.187010 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.186955    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="797af66c-d587-4875-a577-ea9c26502172" containerName="istio-proxy"
Jul 31 15:51:50.187010 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.186963    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5beba8c1-4060-4e95-8f0f-efce040a380a" containerName="main"
Jul 31 15:51:50.187010 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.186970    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2985938c-c483-41a4-acfe-9914ef2641df" containerName="main"
Jul 31 15:51:50.187010 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.186975    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f249c8-4bc7-47f2-ac8c-26247dcb0b77" containerName="main"
Jul 31 15:51:50.187591 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.187057    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="797af66c-d587-4875-a577-ea9c26502172" containerName="istio-proxy"
Jul 31 15:51:50.189684 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.189663    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vkjj8/must-gather-h2klp"
Jul 31 15:51:50.192660 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.192634    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vkjj8\"/\"openshift-service-ca.crt\""
Jul 31 15:51:50.192782 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.192643    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vkjj8\"/\"default-dockercfg-q6zmv\""
Jul 31 15:51:50.192782 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.192693    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vkjj8\"/\"kube-root-ca.crt\""
Jul 31 15:51:50.200143 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.200124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkgpl\" (UniqueName: \"kubernetes.io/projected/d32abf3f-d2ab-4003-8cdb-f473283be3d8-kube-api-access-vkgpl\") pod \"must-gather-h2klp\" (UID: \"d32abf3f-d2ab-4003-8cdb-f473283be3d8\") " pod="openshift-must-gather-vkjj8/must-gather-h2klp"
Jul 31 15:51:50.200237 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.200177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d32abf3f-d2ab-4003-8cdb-f473283be3d8-must-gather-output\") pod \"must-gather-h2klp\" (UID: \"d32abf3f-d2ab-4003-8cdb-f473283be3d8\") " pod="openshift-must-gather-vkjj8/must-gather-h2klp"
Jul 31 15:51:50.204899 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.204879    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vkjj8/must-gather-h2klp"]
Jul 31 15:51:50.301411 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.301345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkgpl\" (UniqueName: \"kubernetes.io/projected/d32abf3f-d2ab-4003-8cdb-f473283be3d8-kube-api-access-vkgpl\") pod \"must-gather-h2klp\" (UID: \"d32abf3f-d2ab-4003-8cdb-f473283be3d8\") " pod="openshift-must-gather-vkjj8/must-gather-h2klp"
Jul 31 15:51:50.301608 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.301459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d32abf3f-d2ab-4003-8cdb-f473283be3d8-must-gather-output\") pod \"must-gather-h2klp\" (UID: \"d32abf3f-d2ab-4003-8cdb-f473283be3d8\") " pod="openshift-must-gather-vkjj8/must-gather-h2klp"
Jul 31 15:51:50.301819 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.301798    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d32abf3f-d2ab-4003-8cdb-f473283be3d8-must-gather-output\") pod \"must-gather-h2klp\" (UID: \"d32abf3f-d2ab-4003-8cdb-f473283be3d8\") " pod="openshift-must-gather-vkjj8/must-gather-h2klp"
Jul 31 15:51:50.311644 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.311618    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkgpl\" (UniqueName: \"kubernetes.io/projected/d32abf3f-d2ab-4003-8cdb-f473283be3d8-kube-api-access-vkgpl\") pod \"must-gather-h2klp\" (UID: \"d32abf3f-d2ab-4003-8cdb-f473283be3d8\") " pod="openshift-must-gather-vkjj8/must-gather-h2klp"
Jul 31 15:51:50.498782 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.498747    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vkjj8/must-gather-h2klp"
Jul 31 15:51:50.629933 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.628483    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vkjj8/must-gather-h2klp"]
Jul 31 15:51:50.630671 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:51:50.630626    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd32abf3f_d2ab_4003_8cdb_f473283be3d8.slice/crio-ba0e2c1957f7cbc9f0fec2cb674d739cab5e9c339c48927b82d46bc685103fe7 WatchSource:0}: Error finding container ba0e2c1957f7cbc9f0fec2cb674d739cab5e9c339c48927b82d46bc685103fe7: Status 404 returned error can't find the container with id ba0e2c1957f7cbc9f0fec2cb674d739cab5e9c339c48927b82d46bc685103fe7
Jul 31 15:51:50.633926 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:50.633906    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:51:51.115253 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:51.115212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vkjj8/must-gather-h2klp" event={"ID":"d32abf3f-d2ab-4003-8cdb-f473283be3d8","Type":"ContainerStarted","Data":"ba0e2c1957f7cbc9f0fec2cb674d739cab5e9c339c48927b82d46bc685103fe7"}
Jul 31 15:51:57.143949 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:57.143912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vkjj8/must-gather-h2klp" event={"ID":"d32abf3f-d2ab-4003-8cdb-f473283be3d8","Type":"ContainerStarted","Data":"3993f4c706e9ee88d1b2820d3f3c9079d5871f0550c4a8dedd082a60e3e3bee5"}
Jul 31 15:51:57.143949 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:57.143958    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vkjj8/must-gather-h2klp" event={"ID":"d32abf3f-d2ab-4003-8cdb-f473283be3d8","Type":"ContainerStarted","Data":"c65bba87b69fcccd20c5128003e90e296668f426b882a45cac687fbb0dfd3947"}
Jul 31 15:51:57.165048 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:51:57.164994    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vkjj8/must-gather-h2klp" podStartSLOduration=1.043894834 podStartE2EDuration="7.164977544s" totalImagesPullingTime="6.12108271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:51:57.16279289 +0000 UTC m=+3815.172004760" watchObservedRunningTime="2026-07-31 15:51:57.164977544 +0000 UTC m=+3815.174189414"
Jul 31 15:52:12.775019 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:12.774982    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-vqq7v_9020da08-566c-41f1-bf8b-339a5d8ed640/istio-proxy/0.log"
Jul 31 15:52:12.815739 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:12.815697    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-vqq7v_9020da08-566c-41f1-bf8b-339a5d8ed640/istio-proxy/1.log"
Jul 31 15:52:13.687670 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:13.687594    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-vqq7v_9020da08-566c-41f1-bf8b-339a5d8ed640/istio-proxy/0.log"
Jul 31 15:52:13.723899 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:13.723862    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-vqq7v_9020da08-566c-41f1-bf8b-339a5d8ed640/istio-proxy/1.log"
Jul 31 15:52:14.765087 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:14.765047    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-nc4f9_5d2cbc9d-b1de-45c3-bbf7-6af3b67d8b4f/authorino/0.log"
Jul 31 15:52:14.809286 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:14.809253    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-kwzdp_2b4e9d1a-343e-45a6-b738-02afc1fa3a1d/kuadrant-console-plugin/0.log"
Jul 31 15:52:20.241396 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:20.241334    2578 generic.go:356] "Generic (PLEG): container finished" podID="d32abf3f-d2ab-4003-8cdb-f473283be3d8" containerID="c65bba87b69fcccd20c5128003e90e296668f426b882a45cac687fbb0dfd3947" exitCode=0
Jul 31 15:52:20.241818 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:20.241407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vkjj8/must-gather-h2klp" event={"ID":"d32abf3f-d2ab-4003-8cdb-f473283be3d8","Type":"ContainerDied","Data":"c65bba87b69fcccd20c5128003e90e296668f426b882a45cac687fbb0dfd3947"}
Jul 31 15:52:20.241818 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:20.241718    2578 scope.go:117] "RemoveContainer" containerID="c65bba87b69fcccd20c5128003e90e296668f426b882a45cac687fbb0dfd3947"
Jul 31 15:52:20.426359 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:20.426311    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vkjj8_must-gather-h2klp_d32abf3f-d2ab-4003-8cdb-f473283be3d8/gather/0.log"
Jul 31 15:52:24.005890 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:24.005850    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kg6bm_7fae56cb-8b98-4d8b-a508-62dac7aa73d1/global-pull-secret-syncer/0.log"
Jul 31 15:52:24.113882 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:24.113845    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f4ptw_78ef4e44-0c0a-4875-9328-e83e1670ca8e/konnectivity-agent/0.log"
Jul 31 15:52:24.172803 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:24.172766    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-32.ec2.internal_17290c901fea2fac12a7cbe3399d875c/haproxy/0.log"
Jul 31 15:52:25.913534 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:25.913496    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-vkjj8/must-gather-h2klp"]
Jul 31 15:52:25.914014 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:25.913697    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-vkjj8/must-gather-h2klp" podUID="d32abf3f-d2ab-4003-8cdb-f473283be3d8" containerName="copy" containerID="cri-o://3993f4c706e9ee88d1b2820d3f3c9079d5871f0550c4a8dedd082a60e3e3bee5" gracePeriod=2
Jul 31 15:52:25.915660 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:25.915633    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-vkjj8/must-gather-h2klp"]
Jul 31 15:52:25.916027 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:52:25.916002    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-h2klp\" is forbidden: User \"system:node:ip-10-0-133-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-vkjj8\": no relationship found between node 'ip-10-0-133-32.ec2.internal' and this object" podUID="d32abf3f-d2ab-4003-8cdb-f473283be3d8" pod="openshift-must-gather-vkjj8/must-gather-h2klp"
Jul 31 15:52:26.145797 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.145769    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vkjj8_must-gather-h2klp_d32abf3f-d2ab-4003-8cdb-f473283be3d8/copy/0.log"
Jul 31 15:52:26.147243 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.147220    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vkjj8/must-gather-h2klp"
Jul 31 15:52:26.149676 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:52:26.149649    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-h2klp\" is forbidden: User \"system:node:ip-10-0-133-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-vkjj8\": no relationship found between node 'ip-10-0-133-32.ec2.internal' and this object" podUID="d32abf3f-d2ab-4003-8cdb-f473283be3d8" pod="openshift-must-gather-vkjj8/must-gather-h2klp"
Jul 31 15:52:26.264452 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.264423    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vkjj8_must-gather-h2klp_d32abf3f-d2ab-4003-8cdb-f473283be3d8/copy/0.log"
Jul 31 15:52:26.264805 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.264781    2578 generic.go:356] "Generic (PLEG): container finished" podID="d32abf3f-d2ab-4003-8cdb-f473283be3d8" containerID="3993f4c706e9ee88d1b2820d3f3c9079d5871f0550c4a8dedd082a60e3e3bee5" exitCode=143
Jul 31 15:52:26.264874 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.264843    2578 scope.go:117] "RemoveContainer" containerID="3993f4c706e9ee88d1b2820d3f3c9079d5871f0550c4a8dedd082a60e3e3bee5"
Jul 31 15:52:26.264935 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.264843    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vkjj8/must-gather-h2klp"
Jul 31 15:52:26.267780 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:52:26.267755    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-h2klp\" is forbidden: User \"system:node:ip-10-0-133-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-vkjj8\": no relationship found between node 'ip-10-0-133-32.ec2.internal' and this object" podUID="d32abf3f-d2ab-4003-8cdb-f473283be3d8" pod="openshift-must-gather-vkjj8/must-gather-h2klp"
Jul 31 15:52:26.272183 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.272168    2578 scope.go:117] "RemoveContainer" containerID="c65bba87b69fcccd20c5128003e90e296668f426b882a45cac687fbb0dfd3947"
Jul 31 15:52:26.285413 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.285394    2578 scope.go:117] "RemoveContainer" containerID="3993f4c706e9ee88d1b2820d3f3c9079d5871f0550c4a8dedd082a60e3e3bee5"
Jul 31 15:52:26.285665 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:52:26.285644    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3993f4c706e9ee88d1b2820d3f3c9079d5871f0550c4a8dedd082a60e3e3bee5\": container with ID starting with 3993f4c706e9ee88d1b2820d3f3c9079d5871f0550c4a8dedd082a60e3e3bee5 not found: ID does not exist" containerID="3993f4c706e9ee88d1b2820d3f3c9079d5871f0550c4a8dedd082a60e3e3bee5"
Jul 31 15:52:26.285741 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.285680    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3993f4c706e9ee88d1b2820d3f3c9079d5871f0550c4a8dedd082a60e3e3bee5"} err="failed to get container status \"3993f4c706e9ee88d1b2820d3f3c9079d5871f0550c4a8dedd082a60e3e3bee5\": rpc error: code = NotFound desc = could not find container \"3993f4c706e9ee88d1b2820d3f3c9079d5871f0550c4a8dedd082a60e3e3bee5\": container with ID starting with 3993f4c706e9ee88d1b2820d3f3c9079d5871f0550c4a8dedd082a60e3e3bee5 not found: ID does not exist"
Jul 31 15:52:26.285741 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.285708    2578 scope.go:117] "RemoveContainer" containerID="c65bba87b69fcccd20c5128003e90e296668f426b882a45cac687fbb0dfd3947"
Jul 31 15:52:26.285951 ip-10-0-133-32 kubenswrapper[2578]: E0731 15:52:26.285931    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c65bba87b69fcccd20c5128003e90e296668f426b882a45cac687fbb0dfd3947\": container with ID starting with c65bba87b69fcccd20c5128003e90e296668f426b882a45cac687fbb0dfd3947 not found: ID does not exist" containerID="c65bba87b69fcccd20c5128003e90e296668f426b882a45cac687fbb0dfd3947"
Jul 31 15:52:26.286002 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.285964    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65bba87b69fcccd20c5128003e90e296668f426b882a45cac687fbb0dfd3947"} err="failed to get container status \"c65bba87b69fcccd20c5128003e90e296668f426b882a45cac687fbb0dfd3947\": rpc error: code = NotFound desc = could not find container \"c65bba87b69fcccd20c5128003e90e296668f426b882a45cac687fbb0dfd3947\": container with ID starting with c65bba87b69fcccd20c5128003e90e296668f426b882a45cac687fbb0dfd3947 not found: ID does not exist"
Jul 31 15:52:26.305464 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.305434    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d32abf3f-d2ab-4003-8cdb-f473283be3d8-must-gather-output\") pod \"d32abf3f-d2ab-4003-8cdb-f473283be3d8\" (UID: \"d32abf3f-d2ab-4003-8cdb-f473283be3d8\") "
Jul 31 15:52:26.305576 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.305530    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkgpl\" (UniqueName: \"kubernetes.io/projected/d32abf3f-d2ab-4003-8cdb-f473283be3d8-kube-api-access-vkgpl\") pod \"d32abf3f-d2ab-4003-8cdb-f473283be3d8\" (UID: \"d32abf3f-d2ab-4003-8cdb-f473283be3d8\") "
Jul 31 15:52:26.307822 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.307798    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32abf3f-d2ab-4003-8cdb-f473283be3d8-kube-api-access-vkgpl" (OuterVolumeSpecName: "kube-api-access-vkgpl") pod "d32abf3f-d2ab-4003-8cdb-f473283be3d8" (UID: "d32abf3f-d2ab-4003-8cdb-f473283be3d8"). InnerVolumeSpecName "kube-api-access-vkgpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:52:26.313199 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.313173    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32abf3f-d2ab-4003-8cdb-f473283be3d8-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "d32abf3f-d2ab-4003-8cdb-f473283be3d8" (UID: "d32abf3f-d2ab-4003-8cdb-f473283be3d8"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:52:26.406949 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.406907    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkgpl\" (UniqueName: \"kubernetes.io/projected/d32abf3f-d2ab-4003-8cdb-f473283be3d8-kube-api-access-vkgpl\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:52:26.406949 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.406942    2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d32abf3f-d2ab-4003-8cdb-f473283be3d8-must-gather-output\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Jul 31 15:52:26.489631 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:26.489597    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d32abf3f-d2ab-4003-8cdb-f473283be3d8" path="/var/lib/kubelet/pods/d32abf3f-d2ab-4003-8cdb-f473283be3d8/volumes"
Jul 31 15:52:29.131885 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:29.131843    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-nc4f9_5d2cbc9d-b1de-45c3-bbf7-6af3b67d8b4f/authorino/0.log"
Jul 31 15:52:29.238548 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:29.238509    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-kwzdp_2b4e9d1a-343e-45a6-b738-02afc1fa3a1d/kuadrant-console-plugin/0.log"
Jul 31 15:52:31.563877 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:31.563800    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tgc6f_b4ffa460-2438-48b3-84b8-049537740158/node-exporter/0.log"
Jul 31 15:52:31.603100 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:31.603069    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tgc6f_b4ffa460-2438-48b3-84b8-049537740158/kube-rbac-proxy/0.log"
Jul 31 15:52:31.636489 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:31.636456    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tgc6f_b4ffa460-2438-48b3-84b8-049537740158/init-textfile/0.log"
Jul 31 15:52:33.877953 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:33.877909    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"]
Jul 31 15:52:33.878514 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:33.878282    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d32abf3f-d2ab-4003-8cdb-f473283be3d8" containerName="gather"
Jul 31 15:52:33.878514 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:33.878301    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d32abf3f-d2ab-4003-8cdb-f473283be3d8" containerName="gather"
Jul 31 15:52:33.878514 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:33.878335    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d32abf3f-d2ab-4003-8cdb-f473283be3d8" containerName="copy"
Jul 31 15:52:33.878514 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:33.878343    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d32abf3f-d2ab-4003-8cdb-f473283be3d8" containerName="copy"
Jul 31 15:52:33.878514 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:33.878440    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d32abf3f-d2ab-4003-8cdb-f473283be3d8" containerName="copy"
Jul 31 15:52:33.878514 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:33.878453    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d32abf3f-d2ab-4003-8cdb-f473283be3d8" containerName="gather"
Jul 31 15:52:33.882690 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:33.882665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:33.885446 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:33.885426    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bs68h\"/\"kube-root-ca.crt\""
Jul 31 15:52:33.885553 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:33.885461    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bs68h\"/\"openshift-service-ca.crt\""
Jul 31 15:52:33.885553 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:33.885461    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bs68h\"/\"default-dockercfg-85wq2\""
Jul 31 15:52:33.890523 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:33.890500    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"]
Jul 31 15:52:33.966720 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:33.966675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqgxj\" (UniqueName: \"kubernetes.io/projected/692ba091-be0b-4f5f-be51-d3dd8b2f72a3-kube-api-access-mqgxj\") pod \"perf-node-gather-daemonset-ngt6m\" (UID: \"692ba091-be0b-4f5f-be51-d3dd8b2f72a3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:33.966921 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:33.966728    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/692ba091-be0b-4f5f-be51-d3dd8b2f72a3-lib-modules\") pod \"perf-node-gather-daemonset-ngt6m\" (UID: \"692ba091-be0b-4f5f-be51-d3dd8b2f72a3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:33.966921 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:33.966748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/692ba091-be0b-4f5f-be51-d3dd8b2f72a3-podres\") pod \"perf-node-gather-daemonset-ngt6m\" (UID: \"692ba091-be0b-4f5f-be51-d3dd8b2f72a3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:33.966921 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:33.966808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/692ba091-be0b-4f5f-be51-d3dd8b2f72a3-sys\") pod \"perf-node-gather-daemonset-ngt6m\" (UID: \"692ba091-be0b-4f5f-be51-d3dd8b2f72a3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:33.966921 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:33.966867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/692ba091-be0b-4f5f-be51-d3dd8b2f72a3-proc\") pod \"perf-node-gather-daemonset-ngt6m\" (UID: \"692ba091-be0b-4f5f-be51-d3dd8b2f72a3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:34.067901 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:34.067855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqgxj\" (UniqueName: \"kubernetes.io/projected/692ba091-be0b-4f5f-be51-d3dd8b2f72a3-kube-api-access-mqgxj\") pod \"perf-node-gather-daemonset-ngt6m\" (UID: \"692ba091-be0b-4f5f-be51-d3dd8b2f72a3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:34.068105 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:34.067918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/692ba091-be0b-4f5f-be51-d3dd8b2f72a3-lib-modules\") pod \"perf-node-gather-daemonset-ngt6m\" (UID: \"692ba091-be0b-4f5f-be51-d3dd8b2f72a3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:34.068105 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:34.067945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/692ba091-be0b-4f5f-be51-d3dd8b2f72a3-podres\") pod \"perf-node-gather-daemonset-ngt6m\" (UID: \"692ba091-be0b-4f5f-be51-d3dd8b2f72a3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:34.068105 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:34.067971    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/692ba091-be0b-4f5f-be51-d3dd8b2f72a3-sys\") pod \"perf-node-gather-daemonset-ngt6m\" (UID: \"692ba091-be0b-4f5f-be51-d3dd8b2f72a3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:34.068105 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:34.067994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/692ba091-be0b-4f5f-be51-d3dd8b2f72a3-proc\") pod \"perf-node-gather-daemonset-ngt6m\" (UID: \"692ba091-be0b-4f5f-be51-d3dd8b2f72a3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:34.068105 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:34.068092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/692ba091-be0b-4f5f-be51-d3dd8b2f72a3-sys\") pod \"perf-node-gather-daemonset-ngt6m\" (UID: \"692ba091-be0b-4f5f-be51-d3dd8b2f72a3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:34.068309 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:34.068114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/692ba091-be0b-4f5f-be51-d3dd8b2f72a3-lib-modules\") pod \"perf-node-gather-daemonset-ngt6m\" (UID: \"692ba091-be0b-4f5f-be51-d3dd8b2f72a3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:34.068309 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:34.068114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/692ba091-be0b-4f5f-be51-d3dd8b2f72a3-podres\") pod \"perf-node-gather-daemonset-ngt6m\" (UID: \"692ba091-be0b-4f5f-be51-d3dd8b2f72a3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:34.068309 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:34.068120    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/692ba091-be0b-4f5f-be51-d3dd8b2f72a3-proc\") pod \"perf-node-gather-daemonset-ngt6m\" (UID: \"692ba091-be0b-4f5f-be51-d3dd8b2f72a3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:34.076647 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:34.076626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqgxj\" (UniqueName: \"kubernetes.io/projected/692ba091-be0b-4f5f-be51-d3dd8b2f72a3-kube-api-access-mqgxj\") pod \"perf-node-gather-daemonset-ngt6m\" (UID: \"692ba091-be0b-4f5f-be51-d3dd8b2f72a3\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:34.193023 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:34.192927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:34.319149 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:34.319121    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"]
Jul 31 15:52:34.321755 ip-10-0-133-32 kubenswrapper[2578]: W0731 15:52:34.321730    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod692ba091_be0b_4f5f_be51_d3dd8b2f72a3.slice/crio-592ba117c70bec9b5980abba7db2e4148cb1be061d5b3f8b811540d57e90938b WatchSource:0}: Error finding container 592ba117c70bec9b5980abba7db2e4148cb1be061d5b3f8b811540d57e90938b: Status 404 returned error can't find the container with id 592ba117c70bec9b5980abba7db2e4148cb1be061d5b3f8b811540d57e90938b
Jul 31 15:52:35.299382 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:35.299327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m" event={"ID":"692ba091-be0b-4f5f-be51-d3dd8b2f72a3","Type":"ContainerStarted","Data":"0c56708c038e7178635d402705af1033200e3bdcda1e379b08c619508b146515"}
Jul 31 15:52:35.299770 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:35.299401    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m" event={"ID":"692ba091-be0b-4f5f-be51-d3dd8b2f72a3","Type":"ContainerStarted","Data":"592ba117c70bec9b5980abba7db2e4148cb1be061d5b3f8b811540d57e90938b"}
Jul 31 15:52:35.299770 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:35.299473    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:35.317536 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:35.317490    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m" podStartSLOduration=2.317476981 podStartE2EDuration="2.317476981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 15:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:52:35.31653096 +0000 UTC m=+3853.325742828" watchObservedRunningTime="2026-07-31 15:52:35.317476981 +0000 UTC m=+3853.326688850"
Jul 31 15:52:35.975171 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:35.975137    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rcwl9_4aa6bb07-f6ad-4a76-88e1-35fc595b4986/dns/0.log"
Jul 31 15:52:35.996335 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:35.996301    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rcwl9_4aa6bb07-f6ad-4a76-88e1-35fc595b4986/kube-rbac-proxy/0.log"
Jul 31 15:52:36.082087 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:36.082050    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x2h67_be585c19-1ea7-4e53-8142-da3d9b2bc15a/dns-node-resolver/0.log"
Jul 31 15:52:36.672134 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:36.672093    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f74pb_82144e0f-aab6-4481-ab7f-86a85e384b20/node-ca/0.log"
Jul 31 15:52:37.916941 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:37.916906    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-vqq7v_9020da08-566c-41f1-bf8b-339a5d8ed640/istio-proxy/0.log"
Jul 31 15:52:37.949007 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:37.948972    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-vqq7v_9020da08-566c-41f1-bf8b-339a5d8ed640/istio-proxy/1.log"
Jul 31 15:52:38.497445 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:38.497407    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bkzwx_964d007a-611d-4215-b4ba-8c4da12c2fbb/serve-healthcheck-canary/0.log"
Jul 31 15:52:39.052300 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:39.052264    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2wszh_e7b0e200-3697-4441-8b42-7333685b3cd4/kube-rbac-proxy/0.log"
Jul 31 15:52:39.073294 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:39.073260    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2wszh_e7b0e200-3697-4441-8b42-7333685b3cd4/exporter/0.log"
Jul 31 15:52:39.094861 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:39.094832    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2wszh_e7b0e200-3697-4441-8b42-7333685b3cd4/extractor/0.log"
Jul 31 15:52:41.312383 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:41.312334    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-ngt6m"
Jul 31 15:52:42.256694 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:42.256659    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-56854c75c8-2j2t9_d9c57708-6c5a-4258-9d55-bd3b472e825d/manager/0.log"
Jul 31 15:52:42.304941 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:42.304910    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-g6bzw_75538cda-d1dc-470a-9322-fbbc951ad737/openshift-lws-operator/0.log"
Jul 31 15:52:44.157873 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:44.157837    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-vqq7v_9020da08-566c-41f1-bf8b-339a5d8ed640/istio-proxy/0.log"
Jul 31 15:52:44.206220 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:44.206179    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-vqq7v_9020da08-566c-41f1-bf8b-339a5d8ed640/istio-proxy/1.log"
Jul 31 15:52:44.450565 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:44.450513    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-8588677cfb-krprn_65ecc030-2e12-4227-bdc5-48b75d9bb5a4/manager/0.log"
Jul 31 15:52:44.483137 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:44.483091    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-kzvnv_3793ccfc-bc3d-4858-80d4-c4a56b09a8fc/s3-init/0.log"
Jul 31 15:52:44.498984 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:44.498953    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-kzvnv_3793ccfc-bc3d-4858-80d4-c4a56b09a8fc/download-hf-model/0.log"
Jul 31 15:52:45.501394 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:45.501326    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-vqq7v_9020da08-566c-41f1-bf8b-339a5d8ed640/istio-proxy/0.log"
Jul 31 15:52:45.538608 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:45.538570    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-vqq7v_9020da08-566c-41f1-bf8b-339a5d8ed640/istio-proxy/1.log"
Jul 31 15:52:49.254500 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:49.254466    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-vmvsm_1db937c9-6ed5-422f-abd3-95849fd5f033/migrator/0.log"
Jul 31 15:52:49.274652 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:49.274619    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-vmvsm_1db937c9-6ed5-422f-abd3-95849fd5f033/graceful-termination/0.log"
Jul 31 15:52:50.706482 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:50.706398    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wc8q_d391192c-cdbd-46ce-9098-444df02b6f1b/kube-multus-additional-cni-plugins/0.log"
Jul 31 15:52:50.728992 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:50.728962    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wc8q_d391192c-cdbd-46ce-9098-444df02b6f1b/egress-router-binary-copy/0.log"
Jul 31 15:52:50.754467 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:50.754442    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wc8q_d391192c-cdbd-46ce-9098-444df02b6f1b/cni-plugins/0.log"
Jul 31 15:52:50.777363 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:50.777323    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wc8q_d391192c-cdbd-46ce-9098-444df02b6f1b/bond-cni-plugin/0.log"
Jul 31 15:52:50.798761 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:50.798732    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wc8q_d391192c-cdbd-46ce-9098-444df02b6f1b/routeoverride-cni/0.log"
Jul 31 15:52:50.821650 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:50.821619    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wc8q_d391192c-cdbd-46ce-9098-444df02b6f1b/whereabouts-cni-bincopy/0.log"
Jul 31 15:52:50.842744 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:50.842716    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wc8q_d391192c-cdbd-46ce-9098-444df02b6f1b/whereabouts-cni/0.log"
Jul 31 15:52:51.256290 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:51.256254    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-sxt8b_c3961a7e-b75a-4837-944f-330a900f1b75/kube-multus/0.log"
Jul 31 15:52:51.325037 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:51.325008    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-97p2g_352aa5e1-673c-4ede-a05a-80da4d81d2f9/network-metrics-daemon/0.log"
Jul 31 15:52:51.344674 ip-10-0-133-32 kubenswrapper[2578]: I0731 15:52:51.344653    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-97p2g_352aa5e1-673c-4ede-a05a-80da4d81d2f9/kube-rbac-proxy/0.log"